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ABSTRACT 


Formula  scoring  is  the  systematic  study  of  measurement  statistics 
expressed  as  linear  combinations  of  products  of  item  scores.  The  theory  is 
currently  being  used  to  compute  non-parametric  estimates  of  ability 
distributions,  item  response  functions,  and  option  response  functions.  The 
theory  has  been  used  to  design  algorithms  for  estimating  item  response 
functions  from  adaptive  test  data  (on-line  calibration),  monitoring  and 
correcting  drift  in  observed  score  distributions  for  adaptive  tests  (on-line 
equating),  computing  optimal  tests  for  cheating,  and  combining 
appropriateness  measurement  information  from  several  subtests.  In  this 
paper  a  portion  of  the  theory  is  developed  from  a  few  principles. 
Applications  are  considered  to  the  problems  of  deciding  whether  ability  has 
the  same  distribution  in  two  demographic  groups,  to  finding  latent  class 
models  that  are  equivalent  to  item  response  models,  and  to  controlling  drift 
in  adaptive  testing  programs. 
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FORMULA  SCORING 


BASIC  THEORY  AND  APPLICATIONS 

Preface 

For  several  years,  Bruce  Williams  and  1  have  been  presenting 
applications  of  a  new  approach  to  measurement,  which  we  call  formula 
scoring.  Our  presentations  to  the  annual  ONR  Contractor's  Conferences  have 
been  punctuated  with  the  phrase,  "It  can  be  shown  ...  . "  This  technical 
report  begins  a  series  of  papers  providing  proofs  of  these  claims.  An 
attempt  will  be  made  to  derive  formula  score  theory  from  a  few  basic 
principles . 

This  version  of  the  report  is  being  used  t:o  introduce  graduate  students 
to  the  work  in  our  laboratory.  Very  explicit,  computational  proofs  are 
provided  for  some  basic  results.  A  shorter  version  is  being  prepared  for 
publication.  • 

Tnanks  to  Bruce  Williams  and  Fritz  Drasgow  there  are  many  data-based 

applications'*'  of  formula  scoring,  which  are  now  starting  to  appear  in 

2 

print  .  The  data-based  applications  are  net  suitable  for  motivating  this 
paper  because  Bruce's  programs  use  concepts  that  are  developed  in  later 
papers.  Therefore  an  alternative  way  to  motivate  the  report  had  to  be 
found. 

Three  examples  of  results  that  can  be  obtained  with  the  theory  have 
been  selected  to  motivate  the  theory.  I  don't  think  the  results  would  have 
been  discovered  without  the  theory.  Each  seems  surprising  -  at  least  to  me 
-  and  somewhat  contrary  to  conventional  psychometric  wisdom.  Each  result 
can  be  easily  proven  with  the  theory.  And  each  result  seems  hard  to  prove 
without  reproducing  the  reasoning  in  the  theory. 
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Some  Examples  to  Motivate  the  Theory 

Formula  score  theory  can  be  used  to  derive  some  unexpected,  hopefully 
useful,  consequences  of  the  assumptions  of  item  response  theory.  Three 
examples  follow. 

The  examples  are  valid  for  parametric  and  non-parametric  item  response 
models.  Except  where  noted,  the  results  hold  for  all  "continuous,  one- 
dimensional,  probabilistic  item  response  models  for  bounded  abilities." 

Thus,  item  response  functions  are  permitted  to  have  any  shape,  provided  they 
are  continuous  functions  of  one  variable  with  values  strictly  between  zero 
and  one.  The  cumulative  distribution  of  ability  also  is  permitted  to  have 
any  shape,  provided  there  is  some  -  possibly  very  large  -  interval  such 
that  the  distribution  is  zero  or  one  outside  this  interval. 

Example  One:  Checking  for  ability  distribution  differences 

A  quick  way  to  recognize  ability  distribution  differences  is  to  check 

average  tests  scores.  Thus,  if  girls  on  the  average  have  higher  test  scores 

than  boys  on  an  unbiased  test  it  is  safe  to  conclude  that  ability  is 

distributed  differently  among  girls  and  boys.  The  converse  obviously  is  not 

true  because  very  different  distributions  may  have  same  mean. 

Using  observed  scores  to  check  for  group  ability  differences  is 

believed  to  be  uniquely  uncomplicated  for  the  Rasch  model.  Since  the  number 

right  score  is  a  sufficient  statistic  for  estimating  ability  it  might  be 

expected  that  it  is  possible  to  determine  the  presence  or  absence  of  group 

ability  differences  by  comparing  distributions  of  number  right  score.  This 

(incorrect)  assertion  can  also  be  expressed  as  follows: 

There  is  a  set  of  statistics  Xq,  X^,  . . .  Xn  such  that  the  group 
ability  distributions  are  different  if  and  only  if  at  least  one 
of  the  statistics  has  different  expected  values  among  girls  and 
boys . 


Preface  and  Examples 


page  3 


Here  n  is  the  number  of  items  on  the  test,  and  X.  is  the  statistic  which 

J 

is  one  if  exactly  j  items  were  answered  correctly  and  zero  otherwise. 

The  theory  shows  that  the  Rasch  model  is  not  unique  in  having  a  small 

number  of  diagnostic  statistics.  The  theory  also  shows  what  can  and  cannot 

be  concluded  when  corresponding  pairs  of  expectations  are  equal. 

For  any  item  response  model,  Rasch  model  or  other,  there  is  a  set  of 

statistics  X^,  X^,  . . .  X^  such  that  if  at  least  one  pair  of  corresponding 

expected  values  differ,  then  the  group  ability  distributions  are  different. 

But  if  corresponding  expected  values  are  equal,  then  the  distributions  still 

may  oe  different.  However,  it  can  be  shown  that  no  statistical  test  (using 

only  the  answers  to  the  n  items  for  data)  exists  that  can  demonstrate  the 

difference!  In  particular,  for  a  test  satisfying  the  Rasch  model  if  boys 

and  girls  have  equal  expected  X^'s  >  then  ability  may  be  distributed 

differently  in  the  two  populations,  but  no  analysis  of  test  data  can  be  used 

to  demonstrate  the  diffe.  ence.  Details  follow  the  proof  of  Theorem  One. 

Recall  that  for  the  Rasch  model  each  item  response  function 

P^(t)  -  Prob(correct  answer  for  item  i  |  ability  >=  t) 

-(t-b.)  ml 

can  be  written  in  the  form  P^(t)  =  [1+e  )  for  some  constant  b^  . 

To  avoid  mathematical  digressions  irrelevant  to  the  main  points  of  this 

paper,  it  will  generally  be  assumed  that  for  i^-j  ,  b.^b.  .  Thus  no  two 

*  J 

Rasch  model  items  have  exactly  the  same  item  response  function. 

As  an  example  of  another  model  having  a  small  set  of  diagnostic 
statistics,  consider  the  generalization  of  the  Rasch  model  having  item 
response  functions  given  by  the  following  equation 

P.(t)  -  c.  +  (l-c.)[  1  +  e’a(t_V]_1  . 

1  l  l  1  1 

As  with  the  Rasch  model,  it  will  generally  be  assumed  that  different  items 

have  different  difficulties.  Thus  if  i.-^i  ,  b.^b.  .  For  this  model  J  is 

1  J 
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less  than  or  equal  to  the  number  of  items,  and  X^  can  be  taken  to  be  the 
score  that  is  one  if  item  j  is  answered  correctly  and  zero  otherwise.  (If 
for  some  i^j  ,  b^-b^  ■  then  a  somewhat  more  complicated  set  of  must  be 

used,  but  J  is  still  small.) 

Incidentally,  these  results  are  related  to  the  identifiability  of 


ability  distributions.  Since  different  distributions  can  give  the  same 


vector  of  expected  X^'s  ,  the  ability  distribution  is  not  identifiable, 
even  when  the  item  response  functions  are  completely  specified. 


Example  Two:  How  to  turn  an  item  response  model  for  an  ability  continuum 
into  an  isomorphic  latent  class  model  with  finitely  many  classes 

Suppose  we  are  given  an  item  response  model  with  continuous  item 
response  functions  *  0,1  and  a  continuous  ability  density  f  .  Using  the 
theoretical  results  in  this  paper  it  can  be  shown  that  it  is  possible  to 
select  abilities  t^<t^<  ...  t^  and  numbers  p(tQ),p(t^),  ...  p(t^)  such 
that  for  each  item  response  pattern  u  ,  the  "manifest  probability" 

* 

Prob (Sampling  an  examinee  with  item  response  patttern  u  )  , 


which  is  ordinarily  computed  by  i  tegrating  the  likelihood  function, 

lik(u*|  ability  =  t)  f(t)  dt 


can  be  computed  by  evaluating  the  sum 
J 

E  lik(u  |  ability  -  t.)  p(t.)  . 
j-0  J  J 

For  the  item  response  functions  given  by  the  formulas  in  Example  One,  J 
can  by  set  equal  to  the  number  of  items. 


Since  the  manifest  probabilities  sum  to  one,  E  p(t^)  =  1  •  Thus  if 
p(tj)  >  0  for  j<J  ,  we  have  a  latent  class  model  with  J+l  classes  that  is 
isomorphic  to  the  continuous  latent  trait  model. 
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I  haven't  found  a  simple  proof  based  only  on  the  results  in  this  paper 

of  the  existence  t.  with  p(t.)  >  0  .  However  the  result  also  is  true  and 
J  J 

is  proven  in  next  paper  in  this  series.  In  any  event,  even  when  some  of  the 


p(t.)  are  negative  the  result  seems  able  to  greatly  reduce  computation 

J  V 

times  in  some  applications  noted  below. 


Example  Three:  On-line  equating  or  Simulation  results  without  simulation 
Consider  two  subtests,  say,  word  knowledge  (WK)  and  arithmetic 
reasoning  (AR) ,  of  a  computer  administered  adaptive  test  such  as  the 
adaptive  version  of  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) . 
Suppose  the  item  pool  for  WK  has  just  been  changed  by  introducing  some  new 
items  that  haven't  been  ministered  often  enough  to  highly  motivated 
examinees  to  have  well  estimated  item  response  functions.  To  analyze  and 
control  the  effect  of  the  new  items  on  the  distribution  of  an  observed  score 

A 

0^  we  wish  to  calculate  three  functions,  usually  computed  by  simulation: 

A 

Expectation  {  0^  |  0^,  =  t) 

A 

F2(t  -  Variance  {  0m  |  0^  -  t) 

P(x|t)  -  Prob  {  0m  £  x  |  0yK  =  t)  . 

F^  and  show  how  the  first  two  conditional  moments  of  the  observed 

score  are  affected  by  the  new  items  and  can  be  used  to  make  corrections. 

For  example,  if  F2(-l)  is  observed  to  increase  very  much  when  the  new 
items  replace  easy  old  items  then  countermeasures  such  as  adding  more  easy 
items  can  be  tried.  P(x|t)  provides  the  remaining  moments.  It  can  be  used 

A 

to  predict  how  the  marginal  distribution  of  0.,v  will  be  affected  by  future 

WK 

changes  in  Che  ability  distribution. 

Since  the  item  response  functions  for  the  new  items  are  not  known, 

A 

simulation  is  not  possible.  (When  the  score  is  a  Bayes  mode  or 

maximum  likelihood  ability  estimate,  then  item  parameter  estimates  derived 
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from  small  samples  of  not  highly  motivated  examinees  may  be  used  to  compute 
the  score,  but  such  estimates  are  not  suitable  for  including  in  a 
simulation.)  Thus,  the  following  result  is  of  interest. 

It  is  generally  possible  to  use  the  item  response  functions  for  the  old 
WK  items  to  compute  functions  CQ(t),c^(t),  ...  c^(t)  and  to  sort  examinees 

A 

into  groups  using  only  an  AR  score  0 ^  .  According  to  the  theory,  the 

A 

conditional  expectation  of  0  (computed  froir  item  scores  for  both  old  and 

WK 

new  items)  can  be  calculated  with  the  formula 


A 

Expectation  (  0 |  0WR«-t  )  - 
K 

2  c.  (t)  Expectation  {  9,.v  \  9  _  is  in  the  kth  score  group)  . 

K  WK  AK 

-o 


In  words,  we  use  9 to  group  examinees  and  then  compute  the  conditional 

A  A 

expected  WK  score  as  a  linear  combination  0,^  group  averages.  The  9 ^ 
score  is  computed  using  item  scores  for  both  old  WK  items  and  new  WK  items. 
However,  only  the  well  estimated  old  WK  item  response  functions  are  used  to 
compute  the  coefficients  of  the  linear  combination.  In  this  way  the  effect 
of  introducing  new  items  on  an  observed  score  at  each  ability  level  can  he 
calculated  from  actual  data.  Since  the  method  does  not  use  item  parameter 
estimates  for  the  new  items,  it  is  not  adversely  affected  by  item  parameter 
estimation  error  on  the  new  items. 

A  similar  formula  gives  the  conditional  variance  since  for  the  same 
c .  and  groups 


A2 

Expectation  (  0^1  0^=0  ) 

K 

“  2  c  (t) 

k=0 


Expectation! 


0 . _  is  score  group 


k  )  . 


Finally,  for  the  random  variable  defined  by 
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X 


1  If  0HK  S  X 


0  otherwise 

A 

the  conditional  distribution  of  8  is  given  by 

WK, 

A 

Prob{  0^  <  x  |(?wt,-t)  -  Expectation  X  |  ®WK“C) 


K 

-  2  c,  (t)  Expectation!  X 

k~0 


l^AR  is  in  group  k  } 


The  calculation  of  these  three  conditional  expected  values  illustrates 
a  more  general  result  described  in  the  discussion  of  "quasidens? ties" 
(Section  Two,  below). 


NOTES 


1.  Formula  score  theory  currently  is  being  used  to  compute  non- 
parametric  maximum  likelihood  estimates  of  ability  distributions, 
item  response  functions,  and  option  response  functions.  The  theory 
has  been  used  to  design  algorithms  for  estimating  item  response 
functions  from  adaptive  test  data  without  interrupting  testing  (on¬ 
line  calibration),  to  compute  optimal  tests  for  cheating,  and  to 
combine  appropriateness  measurement  in: conation  from  several 
subtests.  The  theory  yields  measures  of  item  bias  and  test 
dimensionality.  The  theory  seems  to  lead  to  a  tractible,  non- 
parametric,  multidimensional  item  response  theory,  which  is 
currently  being  developed.  The  theory  is  also  being  applied  to  what 
might  be  called  "online  equating,"  i.e.,  monitoring  and  correcting 
changes  in  the  distribution  of  observed  scores  for  an  adaptive  test 
as  the  test's  item  pool  is  replenished. 

2.  Drasgow,  F. ,  Levine,  M.V. ,  Williams,  B.,  McLaughlin,  M.E.,  and 
Candell,  G.L.  Modelling  incorrect  responses  with  multilinear 
formula  score  theory.  Applied  Psychological  Measurement.  In  press, 
1989;  Drasgow,  F. ,  Levine,  M.V. ,  and  McLaughlin,  M.E.  Multitest 
extensions  of  appropriateness  indices.  Applied  Psychological 
Measurement .  accepted  for  publication,  1989. 


Section  One 


Formula  Score  Theory  and  Equivalent  Distributions 

Formula  score  theory  systematically  studies  measurement  statistics 
expressed  as  linear  combinations  of  products  of  item  scores.  The  theory 
begins  with  an  equivalence  relation  on  ability  distributions. 

We  consider  a  fixed  test  of  n  items.  A  pair  of  distributions  F  and 
G  are  defined  to  be  equivalent  relative  to  the  test  if  every  statistic 
computed  from  the  test's  item  scores  has  the  same  distribution  under  the 
hypothesis 

HqI  Ability  has  cumulative  distribution  F 
as  under  the  alternative  hypothesis 

H^:  Ability  has  cumulative  distribution  G  . 

Notice  that  there  is  no  way  whatsoever  to  use  item  responses  on  the 
test  being  analyzed  to  distinguish  between  a  pair  of  equivalent 
distributions.  For  if  F  is  equivalent  to  C  and  if  the  statistic  X  is 
used  for  hypothesis  testing,  then  decisions  based  on  X  will  be  no  more 
valid  than  decisions  based  on  the  flip  of  a  coin  or  other  irrelevant  random 
process . 

Notice  also  that  equivalence  is  defined  relative  to  a  fixed  test  of 
specified  items.  Thus  a  pair  of  distributions  may  be  equivalent  relative  to 
the  test,  but  distinguishable  if  one  more  item  is  added  to  the  test.  In 
fact,  if  one  of  the  items  is  replaced  by  a  slightly  different  item,  the 
equivalence  relation  may  be  changed.  This  is  a  significant  limitation  of 
the  present  algebraic  version  of  the  theory.  Later  papers  on  applications 
use  metric  concepts  to  get  around  this  problem. 


Section  One:  Equivalent  Distributions  and  the  CS 
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The  main  result  of  this  section  is  a  characterization  of  equivalent 
distributions  in  terms  of  the  expected  values  of  finitely  many  statistics. 
Comments  on  implications  and  applications  of  this  result  are  at  the  end  of 
this  section. 


Item  Response  Theory  and  Formula  Score  Theory 

To  make  the  paper  more  nearly  self-contained  and  to  make  explicit  just 
what  assumptions  of  item  response  theory  are  used  to  prove  the  new  results, 
we  begin  with  some  definitions  from  item  response  theory. 

An  item  response  model  provides  a  probability  measure  for  set  (a)  , 
which  is  interpreted  as  a  set  of  possible  or  actual  examinees.  There  are 
two  types  of  random  variables  in  item  response  theory:  observed  item  scores 
u^(a) ,  u^Ca),  ...  un(a)  and  unobserved  abilities  0(a)  .  Item  scores  are 
either  one  or  zero.  "ui(a)=l"  is  interpreted  as  "examinee  a  successfully 
answered  item  i  . " 

In  this  paper,  the  abilities  0(a)  are  numbers.  However,  after  some 
routine  changes,  all  of  the  results  in  this  paper  and  their  proofs 
generalize  to  multidimensional  abilities,  i.e.,  vector-valued  0( a)'s  . 

Item  response  theory  relates  item  scores  to  abilities  with  functions 
called  item  response  functions 

P^(t)  =  Prob{  Uj=l[0=t  )  . 

P^(t)  is  interpreted  as  the  probability  of  observing  u^(a)  -  1  ,  when 
examinee  a  is  sampled  from  all  those  with  ability  t  . 

In  this  paper,  details  about  the  item  response  functions  are  generally 
left  unspecified.  Only  continuity  and  a  weak  condition,  0  <  P^(t)  <1  ,  are 
assumed.  These  conditions  are  also  implied  by  the  parametric  formulas  of 
most  item  response  models. 
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Formula  scoring  differs  from  much  of  item  response  theory  on  the  domain 
of  definition  of  the  item  response  functions.  In  item  response  models 
P^(i)  is  usually  defined  for  all  numbers  t  ,  despite  the  fact  that  the 
models  predict  essentially  the  same  behavior  from  examinees  with  ability  20 
and  20,000  and  despite  the  fact  that  applications  of  the  parametric  models 
usually  proceed  as  if  abilities  were  bounded. 

In  this  section  the  domain  of  definition  of  the  item  response  functions 
can  be  bounded  or  unbounded.  However,  in  the  following  sections  P^(t)  is 
defined  only  for  t  in  an  interval  of  finite  length.  Some  discussion  of 
this  point  is  at  the  end  of  this  section. 

The  main  assumption  of  item  response  theory  is  local  independence .  It 
asserts  that  item  responses  are  conditionally  independent,  i.e.,  for  any 
sequence  of  zeros  and  ones 

*  *  * 


and  any  ability  t 

Prob{  u^u,  &  u„=u0  ...u=u  I  0  -  t}  =  II.  Prob{  u.-=u.|0-t}  . 
1122  nn1  l  ii‘ 

In  item  response  theory  analyses  of  data,  the  item  responses  are 

recorded  and  inferences  are  made  about  9  .  Only  the  item  responses  are 

observed.  Thus  if  the  word  "statistic"  is  to  be  reserved  for  random 

variables  that  are  functions  of  the  observables,  only  functions  of  the  u^ 

are  statistics.  Since  the  range  of  each  u^  is  finite,  every  function  of 

the  u.  is  a  random  variable.  Thus  X  is  a  statistic  if  and  only  if  X 

is  a  function  of  item  scores. 

The  set  of  all  statistics  for  a  test  is  obviously  a  vector  space  since 

a  linear  combination  of  functions  of  item  scores  is  a  function  of  item 

scores.  Since  the  u^  take  on  only  finitely  many  values,  every  statistic 

2 

can  be  written  as  a  polynomial  in  the  item  scores.  In  fact,  since  u^  =  u^ 
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every  statistic  is  a  linear  combination  of  the  following  statistics,  which 
are  called  elementary  formula  scores, 


1 


ul*  u2 ’  ' ‘ • 

U1U2’  uiu3  * 


u 

n 


u  ,  u 
n-1  n 


n 

n  u.  . 
i*=l  1 

Thus  the  elementary  formula  scores,  or  some  subset  of  the  these  scores,  form 
a  basis  for  the  vector  space  of  all  statistics.  Since  there  are  finitely 
many  (  2n  )  elementary  formula  scores,  the  set  of  all  statistics  is  a  finite 
dimensional  vector  space. 

The  regression  function  R^*)  or  conditional  expectation  function  of 
a  statistic  X 

Rx<t)  =  E(X| 0=t) 

expresses  the  conditional  expected  value  of  the  statistic  as  a  function  of 
ability.  Since  every  statistic  is  a  linear  combination  of  the  elementary 
formula  scores,  local  independence  implies  that  each  regression  function  can 
be  written  in  at  least  one  way  as  a  linear  combination  of  the  following 
functions 

1 

P1(t),  ...  Pn(t) 

P1(t)P2(t),  P1(t)P3(t),  ...  Pnl(t)Pn(t) 

n 

n  p  (t)  . 

i=l 
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The  central  concept  of  formula  score  theory  is  the  canonical  space. 

The  canonical  space  (CS)  of  a  test  is  the  vector  space  of  regression 
functions  of  statistics.  Obviously  it  is  the  vector  space  spanned  by  the 
square- free  monomials,  i.e.  the  products  of  item  response  functions  without 
repeated  factors,  listed  above.  Thus,  the  canonical  space  is  a  finite 
dimensional  vector  space  of  continuous ,  real-valued  functions . 

An  Alternative  Characterization  of  Equivalent  Distributions 

Using  the  canonical  space  it  is  possible  to  derive  a  simpler  test  for 
equivalent  distributions.  The  definition  would  have  us  check  the 
distribution  of  every  statistic.  It  will  be  shown  that  only  finitely  many 
statistics  need  to  be  considered  and  that  all  that  needs  to  be  known  about 
each  statistic  is  its  expected  value.  First,  some  notation. 

F  will  be  used  in  all  sections  of  this  paper  to  denote  the  (generally 
unknown)  ability  distribution.  For  any  statistic  X  and  number  x  ,  the 
distribution  function  of  X  evaluated  at  x  can  be  written 

Prob(X<x)  ■=>  J  Prob(X<x| 0=t)dF(t)  . 

If  G  is  F  or  any  other  distribution,  then  the  distribution  of  X 
relative  to  G  evaluated  at  x  will  be  denoted  by  P(x;X,G)  .  Thus 

P(x;X,G)  =  J  P(X<x|0=t)dG(t)  . 

Similarly,  the  expected  value  of  X  and  the  expected  value  of  X  relative 
to  distribution  G  are  denoted  by 

E(X)  =  J  E(X|0-t)dF(t) 

E(X;G)  =  /  E(X| 0=t)dG(t)  . 

Using  this  notation  the  definition  of  equivalent  distributions  given 
earlier  can  be  succinctly  expressed:  Two  distributions  F^  and  F^  are 
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equivalent  if  for  all  statistics  X  and  real  x 

P(x;X,F1)  -  P(x;X,F2)  . 

Theorem  One  is  an  alternative  characterization  of  equivalent 
distributions . 


Theorem  One :  Let  J+l  be  the  dimension  of  the  canonical  space.  Then  there 
are  J  statistics  X^ ,  ,  .  .  .  Xj  such  that  F^  is  equivalent  to  F2 

if  and  only  if 


E(Xj;F1)  -  E(Xj;F2)  for  j-1 . J  . 

Furthermore .  if  Y^,  Y  ,  . . .  Y^  are  any  statistics  with  linearly 
independent  regression  functions,  then  F^  is  equivalent  to  F2  if  and 
only  if  E(Yj ;FX)  -  E(Yj;F2)  for  j=0,  1,  ...  J  . 


Proof:  Let  h^,  . . .  h^  be  a  basis  for  the  canonical  space.  Since  the 
constant  function  is  in  the  CS ,  h^  can  be  taken  to  be  the  constant 
function,  hg(t)  =  1  •  Since  the  tu  are  in  the  CS,  there  are  statistics 
Xj  such  that  hj(t)  "  E(Xj|0-t)  for  0<j<J  .  For  any  statistic  X  and 
real  x  ,  the  regression  function  of  the  indicator  random  variable,  x 


1,  if  X(u, . u  )  <  x 

1  n 

X  S3  « 

.  0,  if  X(ux . u^)  >  x 

is  in  the  canonical  space  and  consequently  can  be  written 


J 

E(x|0»t)  “  2  a  'n.(t)  . 

j-0  3  J 


Therefore  for  i=l,2 


P(x;X,F.)  =  f  S  a  hj(t)dFi(t) 


Since  E(XQ;F1) 


f  1  dFx(t) 


Vj,<Vfi)  ■ 

1  -  E(X0;F2>  . 
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E(Xj;F1)  -  E(X  ;F2)  for  j-1 


. J 


implies  that  and  are  equivalent.  Conversely,  each  can 

written  as  a  sum  of  products  of  the  binary  item  scores, 


X.  -  2  a  v 

1  -1  is  is 

-  i/=l 

where  v^>  v2>  •••  vl/>  •••  v  n  is  an  enumeration  of  the  2n  elementary 

formula  scores.  Since  v  is  either  zero  or  one,  for  i=l  or  2 

v 

E(v  ;F. )  *■  1-P(0 ; v  ,F.)  . 

i/  i  '  ’  v  1. 

Therefore  "F^  is  equivalent  to  F^"  implies 

E(X.  IF,)  -  2  a  E(x  ;F,) 

1  1  v  v  1 

J  v 

“  2  a^[l-P(0;v^,F^)  ] 

v 

“  E<Xj:F2)  • 

Finally,  if  J+l  statistics  have  linearly  independent  regression 

functions  g^  then  for  some  non-singular  (J+l)x(J+l)  matrix  A”(a.^)  , 

g.(»)=S  a.,h,  (•)  .  The  remainder  of  the  proof  follows  routinely  from 
J  k  Jk  k 

E(Yj:F.)  -  2  ajkE(Xk;F.)  for  j=0,  1,  ...  J  and  i-1,2  . 
lc 


Implications  and  Applications 


The  theorem  has  negative  implications  for  distribution  estimation.  We 
have  observed  that  when  J  is  small,  two  distributions  with  clearly 
different  shapes  can  be  equivalent.  As  noted  in  Example  Two  a  discrete 
distribution  on  a  few  points  may  turn  out  to  be  indistinguishable  from  a 
distribution  with  a  continuous  density.  Thus,  even  when  item  response 
functions  are  known,  it  is  not  possible  to  consistently  estimate  the  ability 
distribution  without  additional  assumptions. 
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Note  that  for  some  applications  it  is  valuable  to  know  that  ability 
distributions  are  equivalent.  Returning  to  Example  One  of  the  Prefa  *,  if 
the  ability  distributions  for  boys  and  girls  are  equivalent  relative  to  the 
test,  then  any  selection  procedure  based  on  test  results  is  as  likely  to 
select  a  boy  as  a  girl. 

The  theorem  shows,  as  was  asserted  in  Example  One,  that  by  checking 
finitely  many  pairs  of  expected  values,  a  difference  between  the  ability 
distributions  can  be  demonstrated.  In  Section  3  it  is  shown  that  J  can  be 
small.  For  the  Rasch  model  and  its  generalization,  J  can  be  taken  equal 
to  the  number  of  test  items  and  X.  can  be  taken  to  be  the  ]th  item  score. 
Thus  a  necessary  and  sufficient  condition  for  there  to  be  a  demonstrable 
difference  between  distributions  is  that  there  be  at  least  one  item  on  which 
the  proportion  of  boys  passing  the  item  is  different  from  the  proportion  of 
girls. 

For  other  models  J  can  be  large  and  the  may  be  complicated. 

Models  with  large  J  are  discussed  in  Section  4.  The  task  of  computing  J 

and  X.  is  also  discussed  in  Section  4. 

J 

Example  Two  illustrates  a  second  situation  in  which  distribution 
equivalence  may  have  practical  importance.  In  Example  Two  we  considered 
replacing  an  ability  distribution  having  a  continuous  density  with  a  step 
function  having  finitely  many  s^eps.  The  goal  in  doing  so  was  to  reduce 
integrals  to  sums.  (In  Section  3  a  procedure  for  calculating  the  location 
and  size  of  the  steps  is  described.)  In  optimal  appropriateness 
measurement^"  it  is  necessary  to  integrate  over  ability  to  obtain  a 
uniformly  most  powerful  test  for  cheating  and  other  forms  of  aberrance. 

Even  for  unidimensional  tests  a  great  deal  of  computing  is  required  to 
compute  the  theoretical  manifest  probabilities  in  Example  Two.  For 
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multidimensional  tests  and  "multi-unidimensional"  test  batteries  such  as 
ASVAB  considerably  more  computation  is  required. 

So  far  we  have  successfully  avoided  computing  multiple  integrals  in  our 

analyses  of  test  batteries  in  which  each  subtest  measures  a  different 

2 

ability  by  using  approximations,  xhe  results  in  this  section  indicate  an 

3 

alternative,  more  general  way  to  calculate  probabilities.  Since  an 
integral  must  be  evaluated  for  each  of  thousands  of  examinees  and  since 
multivariate  quadrature  requires  a  lot  of  computation,  replacing  a 
continuous  multivariate  with  an  equivalent  discrete  distribution  on  a  small 
number  of  points  is  very  desireable. 

This  section  is  concluded  with  comments  on  the  issue  of  bounded  and 
unbounded  ability  continua,  which  is  raised  by  Theorem  One. 

Why  Bounded  Abilities 

Sometimes  whatever  is  being  measured  by  a  test  is  intrinsically 
bounded.  Adding  extremely  hard  items  to  a  test  generally  changes  what  is 
being  measured  and  may  cause  a  test  to  fail  to  be  unidimensional.  Thus  a 
calculus  item  is  not  a  very  hard  arithmetic  item  but  an  item  measuring  an 
ability  or  achievement  other  than  what  is  being  measured  by  a  grade  school 
subtraction  test.  At  the  other  extreme,  a  child  totally  ignorant  of 
subtraction  occupies  a  lower  end  point  on  the  measurement  scale. 

Theorem  One  raises  questions  about  the  domain  of  definition  of  the 
and  also  motivates  considering  bounded  continua.  Suppose  that  on  a 
particular  test  no  examinee  has  an  ability  outside  the  interval  [-5,5]  . 

Then  there  can  be  a  pair  of  inequivalcnt  distributions  and  such 

that  F^(t)  -  ^2^^  ^or  lci  —  ^  »  even  though  no  empirical  study  can 
distinguish  between  F^  and  .  This  awkward  situation  can  be  kept  from 
occuring  by  defining  the  item  response  functions  as  functions  of  abilities 
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in  [-5,5]  .  If  the  are  defined  only  for  |t|  £  5  ,  then  the  CS 

becomes  a  set  of  functions  defined  on  an  interval.  Distributions  that  agree 
on  the  interval  will  then  be  equivalent  in  the  sense  of  Theorem  One  as  well 
as  in  the  intuitive  sense.  Thus  by  treating  the  as  functions  of  a 

bounded  variable  the  intuitive  and  technical  meanings  of  "equivalent"  can  be 
brought  closer  together.  Alternatively,  attention  can  be  limited  to  ability 
distributions  that  are  zero  or  one  outside  this  interval.  Both  options  are 
developed  in  the  next  section. 

The  assumption  of  boundedness  turns  out  to  be  very  weak.  In  any 
practical  measurement  situation,  it  can  be  trivially  satisfied  by 
considering  a  very  large  interval,  an  interval  so  large  that  the  probability 
of  sampling  an  examinee  outside  the  interval  for  all  practical  purposes  is 
zero.  For  theoretical  work,  boundedness  can  be  imposed  on  a  test  model  by 
transforming  abilities  without  affecting  the  only  assumptions  being  made 
about  item  response  functions:  continuity  and  0  <  P^(t)  <1  . 


NOTES 

1.  Levine,  M.V.  and  Drasgow,  F. ,  Optimal  Appropriateness  Measurement. 
Psvchometrika .  1989. 

2.  Drasgow,  F. ,  Levine,  M.V.,  and  McLaughlin,  M.E.  Multitest  extensions  of 
appropriateness  indices.  Applied  Psychological  Measurement,  accepted 
for  publication,  1989. 

3.  The  method  can  be  thought  of  as  a  quadrature  technique  developed  for 
evaluating  the  integrals  that  occur  in  psychometric  applications.  The 
selection  of  the  quadrature  points  and  weights  is  discussed  in  Section 
3.  Each  quadrature  formula  is  exact  for  some  set  of  integrands.  The 
new  method  is  exact  for  integrating  functions  in  the  CS. 
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An  Inner  Product  and  Quasidensities 

When  abilities  are  bounded,  the  CS  has  an  inner  product  with  a  simple 
statistical  interpretation.  And  each  distribution  function  can  be  treated 
as  if  it  had  a  continuous  derivative.  This  "derivative,"  the  quasidensity, 
is  the  subject  of  this  section. 

In  the  remainder  of  this  paper  it  will  be  assumed  that  there  are 
numbers  c<d  such  that  Prob  (c  <  9  <  d)  *»  1  .  Item  response  functions 
will  be  treated  as  functions  defined  on  (c,d]  ,  and  the  canonical  space 
will  be  a  set  of  functions  defined  on  [c,d]  .  After  these  changes  are  made 
the  function  <• , •>  defined  on  pairs  of  functions  f,g  in  the  CS  by 

<f,g>  “  f (t)g(t)dt 

becomes  an  inner  product. 

Note  that  when  the  ability  distribution  has  a  density  and  this  density 
is  in  the  CS,  then  the  inner  product  has  a  statistical  interpretation.  For 
if  R(t)  -  E(X|0“t)  is  the  regression  function  of  a  statistic  X  and  if 
the  ability  distribution  has  a  density  f  also  in  the  CS ,  then  <R,f>  is 
the  expectation  of  X  .  The  major  result  of  this  section  is  to  generalize 
this  property  to  situations  in  which  the  ability  density  is  not  in  the  CS 
and  to  situations  in  which  the  ability  distribution  is  not  differentiable. 

It  will  be  shown  that  there  is  a  unique  continuous  function  g  in  the  CS 
such  that  for  all  statistics  X 

E(X)  =  SdQ  E (X  |  0=t)  dF(t) 

-  E(X| 0=t)  g(t)dt 

-  <RX>S>  • 

Theorem  Two:  If  P(c<0<d)  =  1  ,  then  there  is  a  unique  continuous  function 
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g  in  the  CS  such  that  for  every  statistic  X 

E(X)  =  E(X|*-t>  g(t)dt  . 

Proof :  Let  h^,  h^,  . . .  hj  be  an  orthonormal  basis  for  the  CS  relative  to 

its  inner  product  <• , •>  .  Thus  <h^,h^>  -  1  or  zero  according  to  whether 

i  j  .  For  each  j<J  a  statistic  X^  can  be  found  such  that  E(Xj|0”t) 

-  h.(t)  because  every  function  in  the  CS  is  the  regression  function  of  at 

least  one  statistic.  Let  X  be  any  statistic  and  R  its  regression 

function.  Since  the  h.  form  a  basis  for  the  CS,  R  can  be  written 

J  x 

¥•>  -  W> 

for  some  constants  b.  .  Since  the  h.  are  orthonormal  <R,,  ,h.>  ■=»  b.  and 

J  J  *  J  J 

¥•)  -2j  • 


Consequently 


E(X)  -  J*  R^t)  dF(t) 

-  J*  Sj  <Rx,hj>hj(t)  dF(t) 

-  Sj  <Rx.hj>  J*  h  (t)dF(t) 

-  2L  <Rx,hj>E(Xj) 

-  2.  /J  Rx(t)h.(t)dt  E(X  ) 

-  /J  ^(t)  S.  E(X  )h  (t)dt 


E(X|*-t)g(t)dt 


for  g  “  2  E(Xj)hj(*)  in  the  CS. 

To  prove  uniqueness ,  suppose  that  for  some  h  in  the  CS 


E(X)  -  fc  Rx(t)h(t)dt 


for  all  statistics  X  .  Since  the  h.  form  a  basis,  h(-)  =  E  a.h.(')  for 

J  J  J 
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some  constants  a.  .  Since  the  h.  are  orthonormal,  for  X  -  X. 

J  J  J 

E(X  )  -  (t)h(t)dt 

J  j 

“  /c  hj(t)  ^  akhk(t)dt 

-  S  a.  <h .  ,h,  > 
k  R  J  k 

"  a .  . 

J 

Thus  h=g  ,  as  was  to  be  proven. 


If  G=F  or  any  other  distribution  function,  then  G  will  be  called  a 
distribution  on  [c,d]  if  for  t<c  ,  G(d)  -  G(t)  ■=  1  .  If  G  is  F  or 
any  other  distribution  on  [c,d]  then  a  function  g  in  the  canonical  space 
is  called  the  quasidensity *  for  G  if  for  all  statistics  X 

E(X;G)  -  Jd  E(X| 0~t)  g(t)dt  . 

Note  that  Theorem  Two  implies  that  every  distribution  on  [c,d]  has  a 
unique  quasidensity.  Furthermore  the  proof  shows  that  the  quasidensity  for 
G  can  be  written  as 


g(0  -  2  E(X  ;G)h  (.) 
j-0  J  J 

where  {h.)d  n  is  any  orthonormal  basis  for  the  CS  and  each  X.  satisfies 
J  J 

Rjj  •=  h.  .  Since  the  quasidensity  is  unique,  the  choice  of  the  orthonormal 

j  J 

basis  and  statistics  X^  used  in  the  formula  is  inconsequential. 

At  the  end  of  this  section  some  facts  about  quasidensity  densities  are 

listed  and  proven.  The  quasidensity  for  the  unit  step  at  -1  is  shown  to 

have  the  simple  form  g(t)  =  S  h.(-l)h.(t)  where  {h.}'?  _  is  any 

j<j  J  J  j  J”*-1 

orthonormal  basis  for  the  CS.  This  formula  was  used  to  compute  an 
approximation  to  the  quasidensity  for  the  unit  step  at  -1  .  The  first  19 
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hj's  for  100  three  parameter  logistic  items  by  the  methods  in  Section  4. 

Figure  One  shows  the  graph  of  q(t)  =  2  h.(-l)h.(t)  .  If  q(t)  is 

j<18  J  J 

multiplied  times  any  of  the  100  logistic  functions  and  integrated,  the 

d 

result  should  be  very  close  to  P^(-l)  .  |P^(-1)  -  /  P^(t)q(t)  dt|  was 

found  to  be  generally  small,  as  shown  in  Table  One. 

For  shorter  tests,  the  quasidensity  of  the  unit  step  function  can  be 
computed  without  approximation.  The  graph  shown  in  Figure  One  is  typical. 

The  precision  of  the  approximation  shown  in  Table  One  serves  to 
illustrate  a  point  developed  in  Section  Four:  For  some  purposes,  high 
dimensional  canonical  spaces  can  be  approximated  by  much  lower  dimensional 


spaces . 
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Figure  One:  Cumulative  distribution  function  for  the  unit  step  function  at 


6  =  -1  and  its  quasidensity 
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Table  One:  P^(-l)  and  an  Approximation 


item 

P^-D 

JPjq 

diff 

1. 

.1223 

.1223 

.0000 

3. 

.0852 

.0852 

.0000 

5. 

.1449 

.1449 

.0000 

7. 

.2958 

.2958 

.0000 

9. 

.2601 

.2601 

.0000 

11. 

.2380 

.2380 

.0000 

13. 

.3024 

.3023 

.0001 

15. 

.3385 

.3385 

.0000 

17. 

.2798 

.2795 

.0003 

19. 

.4482 

.4483 

.0000 

21. 

.7634 

.7634 

.0000 

23. 

.7804 

.7804 

.0000 

25. 

.8695 

.8696 

.0000 

27. 

.2832 

.2832 

.0000 

29. 

.1463 

.1463 

.0000 

31. 

.1396 

.1396 

.0000 

33. 

.2578 

.2578 

.0000 

35. 

.2262 

.2262 

.0000 

37. 

.2521 

.2521 

.0000 

39. 

.3256 

.3256 

.0000 

41. 

.3734 

.3734 

.0000 

43. 

.6150 

.6149 

.0001 

45. 

.7948 

.7948 

.0001 

47. 

.7835 

.7835 

.0000 

49. 

.8228 

.8227 

.0001 

51. 

.1133 

.1133 

.0000 

53. 

.0605 

.0605 

.0000 

55. 

.2024 

.2024 

.0000 

57. 

.3809 

.3809 

.0000 

59. 

.3495 

.3495 

.0000 

61. 

.1521 

.1521 

.0000 

63. 

.2931 

.2931 

.0000 

65. 

.2569 

.2569 

.0000 

67. 

.4459 

.4459 

.0000 

69. 

.6179 

.6179 

.0000 

71. 

.7718 

.7718 

.0000 

73. 

.8167 

.8167 

.0000 

75. 

.8775 

.8774 

.0000 

77. 

.2074 

.2074 

.0000 

79. 

.0660 

.0660 

.0000 

81. 

.2858 

.2858 

.0000 

83. 

.1365 

.1365 

.0000 

85. 

.2095 

.2095 

.0000 

87. 

.2888 

.2888 

.0000 

89. 

.3565 

.3565 

.0000 

91. 

.3742 

.3742 

.0000 

93. 

.7894 

.7894 

.0000 

95. 

.7856 

.7856 

.0000 

97. 

.8536 

.8532 

.0004 

99. 

.8159 

.8159 

.0000 

item 

P.(-D 

JP^ 

diff 

2. 

.0601 

.0601 

.0000 

4. 

.1639 

.1639 

.0000 

6. 

.1878 

.1878 

.0000 

8. 

.2058 

.2058 

.0000 

10. 

.3345 

.3345 

.0000 

12. 

.2093 

.2093 

.0000 

14. 

.2965 

.2965 

.0000 

16. 

.4869 

.4869 

.0000 

18. 

.7576 

.7575 

.0001 

20. 

.8665 

.8665 

.0000 

22. 

.  9014 

.9012 

.0002 

24. 

.9054 

.9054 

.0000 

26. 

.1391 

.1391 

.0000 

28. 

.2334 

.2334 

.0000 

30. 

.1504 

.1504 

.0000 

32. 

.1374 

.1374 

.0000 

34. 

.2314 

.2313 

.0001 

36. 

.1881 

.1880 

.0000 

38. 

.2788 

.2788 

.0001 

40. 

.2676 

.2673 

.0003 

42. 

.5322 

.5322 

.0000 

44. 

.6617 

.6614 

.0003 

46. 

.7852 

.7851 

.0001 

48. 

.8159 

.8159 

.0000 

50. 

.9064 

.9062 

.0001 

52. 

.0662 

.0662 

.0000 

54. 

.2013 

.2013 

.0000 

56. 

.2697 

.2697 

.0000 

58. 

.1809 

.1809 

.0000 

60. 

.3370 

.3370 

.0000 

62. 

.2812 

.2812 

.0000 

64. 

.2673 

.2673 

.0000 

66. 

.3876 

.3876 

.0000 

68. 

.6903 

.6903 

.0000 

70. 

.8457 

.8454 

.0003 

72. 

.7427 

.7427 

.0000 

74. 

.8800 

.8800 

.0000 

76. 

.1406 

.1406 

.0000 

78. 

.2022 

.2022 

.0000 

80. 

.2454 

.2454 

.0000 

82. 

.0996 

.0996 

.0000 

84. 

.1368 

.1368 

.0001 

86. 

.1741 

.1740 

.0000 

88. 

.2685 

.2684 

.0001 

90. 

.4457 

.4457 

.0000 

92. 

.3632 

.3632 

.0000 

94. 

.4970 

.4970 

.0000 

96. 

.7681 

.7681 

.0000 

98. 

.7984 

.7984 

.0000 

100. 

.9671 

.9674 

.0003 

Averages:  .414162  .414137  .000025 
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An  Application  of  Quasidensities 
2 

As  an  illustrative  application  ,  we  return  to  Example  Three  of  the 

A 

Preface.  Let  X  be  a  statistic  such  as  9  for  which  we  desire 

Wi\ 

E(X|0“t)  .  Let  be  binary  random  variables  indicating  group 

membership.  For  example  in  Example  Three,  K  numbers  x^  in  the  range  of 

A 

9 can  be  used  to  define  variables  of  the  form 
AR 

A 

M,  =  1  if  1 9 -  x.  I  <  .  5  ,  else  zero 
k  1  AR  k1 

dividing  examinees  into  K  not  necessarily  disjoint  groups.  Let 

q. ,  . . .  q„  be  the  quasidensities  for  the  (conditional)  distributions 
i  K 

Fk(t)  =  Prob  (  $  <  t  |  Mk  =  1  )  . 

Suppose  K  is  large  enough  and  the  Fk  different  enough  so  that  some 
subset  of  the  qk  forms  a  basis  for  the  CS .  Let  q(*;s)  be  the 
quasidensity  of  the  unit  step  at  s  in  [c,d]  .  Then  there  must  be  numbers 
ck°ck(s)  such  that 

q(t;s)  -  2  c,  (s)q,  (t)  ,  c<t<d  . 

k<K 


From  the  definition  of  q(«;s)  we  have 

E(X|0-s)  -  E(X|0=t)q(s;s)  dt  . 


Thus 


E(X|*-s)  =  E(X|ff-t)  S  c  (s)q  (t)  dt 
C  k<K  K  k 

-  S  c.  (s)  E(X| 0“t)q,  (t)  dt 

k<K  K  C  K 

-  S  c  (s)  E(X|M  =1)  . 
k<K 
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Thus  the  regression  function  on  the  left  -  expressing  a  conditioning  on  an 
unobserved  ability  -  equals  a  linear  combination  expected  values  of  observed 
scores  for  tne  objectively  defined  groups. 

To  apply  this  result  K  is  taken  to  be  large,  q(*;s)  is  computed  with 
the  identity  (derived  at  the  end  of  this  section) 

q( * ; s)  -  2  h  (s)h  (t)  . 
j  J  J 

The  qk  are  estimated  by  maximum  likelihood.  The  cjc(‘)  are  computed 
for  each  s  by  minimizing  a  quadratic  objective  function  such  as 

Q(cr  ...  cR)  *»  (qs(t)  -  2  ck(s)qk(t)  ]2dt  . 

In  this  way  a  conditional  expected  value  of  a  statistic  given  ability  can  be 
computed  when  simulation  is  not  possible  or  practical. 

In  addition  to  the  three  examples  in  Example  Three,  there  is  the 
interesting  special  case  of  X«*un+^  ,  the  item  score  for  a  new  item,  and 

E(X| 0=t)  =  Pn+1(t)  , 

its  item  response  function.  Thus  the  formula  at  the  bottom  of  page  24 
expresses  an  unknown  item  response  function  as  a  linear  combination  of  the 
expected  values  of  statistics. 
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Summary  of  Properties  of  Quasidensities 


CS 


Throughout  this  summary,  let  {h.)j=o 
and  {Xj ) j=Q  be  statistics  satisfying 


be  an  orthonormal  basis  for  the 
E(Xj |0=t)  =  hj (t)  for 


c<t<d  . 


Properties  One,  Two,  and  Three  are  useful  for  guessing  the  shape  of  the 
quasidensity  when  F  has  a  density  in  the  CS  or  is  closely  approximated  by 
a  distribution  on  [c,d]  with  a  density  in  the  CS.  Property  Four  can  be 
used  even  if  no  close  approximation  of  F  has  a  dens  ty  in  the  CS. 

Property  Five  underscores  the  identifiability  of  the  quasidensity  by 
exhibiting  a  strongly  consistent  (albeit,  inefficient)  estimate  for  the 
quasidensity. 


Defining  Property  of  Ouasidensities :  A  function  g  in  the  CS  is  the 
quasidensitv  for  G  if  for  all  statistics  X 

E(X|*-t)  dG(t)  -  J*  E(X|*«t)g(t)  dt 

c  c 

Formula  for  Ouasidensities:  g(t)  =  S.  E[Xj  ;G]h,.  (t) 


Quasidensitv  for  Step  Functions:  Let  Gg  be  the  unit  step  at  s  and 
q(*;s)  its  quasidensitv.  Then 


q(t;s)  -  S.  hj(s)hj(t) 

Proof:  E[X. ;G  ]  -  jf  h.(t)  dG  (t)  -  h.(s) 

J  &  c  j  s  J 

Property  One:  If  G  has  a  continuous  density  G'  and  G'  is  in  the 
canonical  space  then  G'  is  the  quasidensitv  of  G  . 


Proof :  <R^,G'>  =  E(X;G)  for  all  statistics  X  . 

Property  Two:  If  G  has  a  (not  necessarily  continuous)  density  G'  then 


the  quasidensitv  of  G  is  the  projection  of  G'  into  the  canonical 
space  in  the  sense  that  the  quasidensitv  g  is  the  unique  miniroizer  in 
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the  GS  of 


Sdc  [G'(t)  -  g(t)]2dt  . 

Proof :  The  general  function  in  the  CS  can  be  written  h(t,d)  = 

2.[E(X .:G)  -  d.lh.(t)  for  some  vector  of  constants  d  .  Since  E(X.;G)  = 
J  J  J  J  J 

Jd  h.(t)G'(t)dt  and  since  the  h.  are  linearly  independent  it  suffices  to 
c  J  J 

show  that  h(t,0)  is  a  minimizer.  This  follows  from  the  identity 

Jd  (G'(t)  -  h(t,d)]2dt  -  Jd  G'2  -  2  E(X  ;G)2  +  2  d2  . 

c  c  J  J 

Property  Three:  If  distributions  are  close,  then  their  nuasidensities  are 

close  in  the  following  sense: 


If  and  F2  be  distributions  on  [c,d]  with  quasidensities 

and  q2  and  Jd  [F^t)  -  F2(t)]2dt  <  £  ,  then  Jd  t q-j^C t)  -  q2(t)]2dt 
<  c 

Proof:  For  i-1,2  F^  can  be  written  F^  =  q^  +  (F^q^)  ■=*  q^  +  ^  .  For 
any  orthonormal  basis  { h ^ }  ,  <r^,hj>  =  0  for  each  j  .  Thus  fcr  any  h 
in  the  CS,  <r^,h>  -  0  .  Consequently 

Jd  [F1(t)  -  F2(t)]2dt  -  Jd  (q]L(t)  -  q2(t)]2dt 

+  0 

+  Jd  [rx(t)  -  r2(t) ] 2  dt 
>  (qi(t)  -  q2(t)]2dt  . 

Property  Four*  The  nuasidensitv  of  the  limit  of  a  convergent  sequence  of 
distributions  on  (c,d]  is  the  limit  of  the  corresponding  sequence  of 
ouasidensities .  More  precisely. 


If  {G^}  is  a  sequence  of  distribution  functions  on  (c,d)  weakly 
convergent  to  a  distribution  G  on  [c,d],  then  the  sequence  of 
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quasidensities  of  the  converges  uniformly  to  the  quasidensity  of 

G  . 


Proof:  Let  X  be  any  statistic.  Since  the  regression  function  for  X  is 

continuous,  by  Helly's  second  theorem  lim  E(X,G  )  -  lim  J  E(X|0“t)  dG  (t) 

n  3.  n 

~  E(X;G)  .  uniformity  follows  from  the  continuity  of  quasidensities. 

The  ability  distribution  clearly  isn't  determined  by  item  response 
data.  This  is  obvious  from  Theorem  One.  When  J  is  small,  markedly 
different  distributions  can  be  equivalent.  The  quasidensity,  on  the  other 
hand,  can  be  recovered  from  item  response  data.  The  formula  for  the 
quasidensity  shows  that  all  one  needs  to  estimate  the  quasidensity  from  data 
is  the  expected  values  of  finitely  many  statistics. 


Property  Five :  The  quasidensitv  is  determined  by  item  response  data  in  the 
sense  that  there  is  a  strongly  consistent  quasidensitv  estimation 
procedure. 


Proof:  The  variance  of  each  X^  must  be  finite  because  there  are  only 

finitely  many  possible  values  for  X^  ,  one  for  each  of  2n  possible 

response  patterns.  Consequently  X.  „  ,  the  sample  average  for  N 

J 

randomly  sampled  examinees,  tends  to  E(X^)  with  probability  one  as  sample 
size  is  increased.  In  fact,  the  multivariate  strong  law  of  large  numbers 


implies  that  the  vector  of  sample  means  <X 


0  ,N’ 


X  M>  almost  surely 

J  i  IN 


converges  to  the  vector  of  expected  values  <E(Xq) . E(X^)>  .  Since  the 

quasidensity  g  for  the  ability  distribution  F  satisfies 


g(t)  -  S  E(X. )h. (t) 
j-0  3  3 


the  random  function  defined  by 
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%(c)  "  A  xj.n  hj(t)  '  c-“d 

almost  surely  converges  to  the  quasidensity.  Furthermore,  the  convergence 


must  be  uniform  in  t  because  the  h.  are  continuous  on  [c,dl 

J 


NOTES 


1.  The  term  seems  apt  because  the  prefix  "quasi"  means  "to  some 
degree,  in  some  manner."  Although  g(t)  may  be  negative, 

Sdc  h(t)  dG(t)  -  Sdc  h(t)g(t)  dt 

at  least  for  every  function  h  in  the  CS. 


2.  There  is  a  technical  problem  beyond  the  scope  of  this  paper  that 
arises  in  applications  of  this  type.  When  the  CS  has  been  computed 
from  only  a  subset  of  the  test  items  then  Rj,(t)  “  E[X|0=t]  may  not 

be  in  the  CS.  In  this  case  the  analysis  yields  an  estimate  of  the 
projection  of  into  a  subspace  of  the  CS  computed  from  all  the 

test  items.  We  have  observed  that  when  only  a  small  number  of  items 
have  not  been  included  the  projection  and  R^(t)  agree  to  several 

decimals,  provided  the  not  included  items  are  not  extremely  easy, 
extremely  hard  or  otherwise  atypical. 
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The  Canonical  Space 
Logistic  Models  and  the  Examples 


This  section  contains  proofs  and  additional  details  for  assertions  made 
earlier  about  the  examples.  We  begin  the  study  of  computing  the 
dimensionality  of  the  CS  and  selecting  basis  functions  h^  and  statistics 
X.  for  some  simple  models. 


The  Rasch  Model  and  its  Generalization 


In  Examples  One  and  Two  it  was  asserted  that  the  generalization  of  the 
Rasch  Model  has  J  less  than  or  equal  to  the  number  of  items  and  that  the 
item  response  functions  or  some  subset  of  them  form  a  basis. 

If  P^(t)  -  c^  +  (l-c^)[  1  +  e  ^i^]  ^  then  we  can  solve  for  eat 
and  obtain 


at  a^i  P.(t)-c. 

^  “  e  I  1 

l-P.(t) 

Thus  for  ii*j 

ab .  ab . 

e  1  [  1-P  <t)  ][  P-^-c.  j  -  e  J  [  l-P.(t)  ][  P  (t)-c  ]  . 


If  b.*b. 
i  J 


form 


then  this  equation  can  be  simplified  to  obtain  an  expression  of 


P.(t)P.(t)  =  a  +  bP.(t)  +  cP.(t) 
i  j  i  j 

where  a,  b,  and  c  are  independent  of  t  .  Thus  any  product  of  two  item 
response  functions  can  be  rewritten  as  a  linear  combination  of  the  item 
response  functions  plus  a  constant.  Using  this  fact  it's  easy  to  prove  the 
assertions  concerning  these  models  in  Example  One. 
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If  item  response  functions  satisfy  the  formula  for  the  Rasch  model  or  its 


eneralization  with  b.^b.  for  i^i  .  then 

-  L  j  -  j  - 

1.  The  dimensionality  of  the  canonical  space  is  less  than  or  equal  to 


one  plus  the  number  of  items 

2 .  The  constant  function  and  the  item  response  functions  or  some 
subset  of  these  functions  form  a  basis  for  the  CS 

3.  The  item  scores  satisfy  the  condition  on  the  X.  in  Theorem  One 

- -  x  - 

and  Example  One . 

Proof :  Since  the  square-free  monomials  span  the  canonical  space,  it  is 
sufficient  to  show  that  every  square- free  monomial  can  be  expressed  as  a 


linear  combination  of  the  plus  the  constant  function  h^Ct)"*!  .  Any 

square-free  monomial  containing  two  or  more  of  the  item  response  functions 

can  be  written  in  form  RP^Pj  for  i^j  for  R  equal  to  a  square- free 

monomial  not  divisible  by  P.  or  P.  .  Thus  RP.P.  -  aR  +  bRP.  +  cRP.  can 

i  J  i  J  i  J 

be  rewritten  as  the  linear  combination  of  three  square -free  monomials,  each 
of  which  has  fewer  factors  than  the  original  monomial.  By  iterating  this 
process  one  eventually  obtains  a  linear  combination  of  square -free  monomials 
depending  on  one  of  the  P^,  or  none  of  the  P^  (i.e.  h^).  Thus  h^  and 
the  P^  span  the  CS,  which  proves  1.  and  2.  The  remaining  assertion 
follows  from  E(u.J0=t)  ■=■  P^(t)  . 
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Selecting  Points  for  Example  Two 

In  Example  Two  we  considered  changing  integrals  to  sums.  It  was 
asserted  that  there  were  numbers  tQ,t^,  •••  cj  ancl  P(cq).P(c^)>  p(tj) 
such  that  for  any  vector  of  zeros  and  ones  u  ,  the  manifest  or  pattern 


probability 


could  be  written 


X  lik(u  |t)  dF(t) 


\  lik(u  |tk)p(tk)  . 

This  is  an  example  of  a  more  general  result,  proven  in  this  subsection:  For 

any  statistic  a  (including  the  statistic  that  is  one  if  the  observed  item 

* 

response  pattern  equals  u  and  zero  otherwise) 


X  E [ X | 0-t]  dF(t)  -  Sk  E[X|0-tk]p(tk)  . 

The  choice  of  the  tk  and  computation  of  the  p(tk)  is  also  discussed.  We 

use  the  notation  q(*;tk)  for  the  quasidensity  of  the  unit  step  function  at 

t,  and  the  fact  that  q(»;t,)  -  2.  h.(t,  )h.(»)  for  any  orthonormal  basis 
K  K  j  j  k  j 

for  the  CS. 

The  result  need  only  be  proven  for  bounded  ability  continua  since  any 
item  response  model  with  continuous  P^0,1  can  be  transformed  by  an 
invertible  transformation  to  a  bounded  model.  The  proof  is  split  into  two 
parts:  The  existence  of  a  basis  consisting  of  quasidensities  and 
interpretation  of  the  p(tk)  • 

The  results  indicate  the  following  procedure  for  selecting  points  and 
computing  p's  for  a  model  with  CS  having  basis  {h^ ) j_q  : 

1.  Choose  t^.t^,  •••  Pj  such  that  the  matrix 
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o 

rr 

o 

V-/ 

hi<t0) 

••  hJ(t0)  ' 

W 

W  ■ 

..  ^y 

h0^CJ^ 

W  ■ 

..  h^) 

is  nonsingular 

2.  Compute  p(tQ),p(t^),  ...  P(tj)  by  solving  the  linear  equations 
g(tj)  -  2k  p(tk)q(tj ;tk)  for  j-0,1,  ...  J  where  g  is  the 
quasidensity  of  F  . 

For  the  generalization  of  the  Rasch  model,  the  procedure  can  be  simplified; 
the  p(t^)  can  be  found  by  solving  the  system  of  linear  equations 

E(Ui)  ”  2k  P^k^i^V  i“1»  •••  n 
1  "  Sk  P(tk} 

Generalizations  and  proofs  follow. 


If  a  test  has  continuous  item  response  functions  ^0,1  defined  on  an  interval 
[c,d]  then  the  CS  has  a  basis  consisting  of  quasidensities  of  unit  step 
distributions . 


Proof:  Let  (h.l.  -  be  an  orthonormal  basis  for  the  CS  and  let  h(t) 
-  J  J”0 

denote  the  column  vector 


h(t)  -  <hQ(t)  .h^^Ct) ,  ...  hj(t)>T  . 

Since  the  are  linearly  independent  there  must  be  J+l  values  of  t 

such  that  the  vectors  http) ,h(t^) ,  ...  h(tj)  are  linearly  independent.  It 
follows  that  the  partitioned  matrix  (hCt^) ,h(t^) ,  ...  h(tj)]  has  an 
inverse,  say  A=*(a„)  .  Consequently,  using  Kronecker's  delta  notation  each 
h.  can  be  written  as  a  linear  combination  of  the  quasidensities  q(*;t^) 
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-2  a  .  q(t; t  )  . 
m  mj  n  m 

Thus  the  quasidensities  form  a  basis  for  the  CS. 

As  a  corollary,  we  have 

The  quasidensities  of  unit  step  distributions  at  ^q’^I’  *"J  span  the  CS 

if  and  only  if  (h(tg) ,h(t^) ,  ...  h(tj)]  is  non-singular. 

In  practice  on  this  type  of  problem  we  compute  the  t^  recursively. 
After  having  chosen  tQ,t^,  •  ■  •  tk  we  choose  tk+^  such  that  h(tk+^) 
makes  a  relatively  large  angle  with  its  projection  into  the  linear  space 
spanned  by  h(tQ)  .h^) ,  ...  h(tk)  . 

After  the  t^  are  selected  the  calculation  of  the  p(tk)  ■'•s  straight 
forward.  Since  the  quasidensities  for  the  t^  form  a  basis  for  the  CS,  the 
ability  distribution's  quasidensity  is  a  linear  combination  of  the 
q(*;tk)  and  the  coefficients  of  the  combination  are  unique.  The  p(tk) 
are  simply  the  coefficients  of  the  linear  combination. 

Let  { q ( • ; t^) ) k„Q  be  a  basis  for  the  CS  and  the  quasidensitv  for  the 
ability  distribution  be  2^  p(t^)q(  • ;  ti^)  .  Then  for  any  statistic  X  , 

E(X)  -  2^  E(X| 0“tk)p(tk)  .  In  particular  for  anv  vector  of  zeros  and  ones 
u  ,  Prob(u*»u  )  “  2^  Prcb(u=u  |0=tk)p(tk)  . 

Proof :  Let  X  be  any  statistic.  Then  from  the  defining  property  of 


quasidensities 


E(X)  -  J*  E(X| 0=t)  Ek  p(tk)q(t;tk)  dt 
-  2k  p(tk)  Jjj  E(X|^t)q(t;tk)  dt 


-  2k  p(tk)  E(X|^-tk) 

* 

In  particular  for  any  vector  of  zeros  and  ones  u  if  X  is  the  random 
variable  that  is  one  if  u*=u>f  and  zero  otherwise,  Prob(u=u^)  =  E(X) 
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-  2k  Prob(u-u*|0-tk)p(tk)  . 

Models  with  Very  Large  J 

If  J  is  small,  as  is  the  case  with  the  Rasch  model  and  its 
generalization,  then  standard  techniques  can  be  used  for  computing  an 
orthonormal  basis.  ^ever,  if  the  dimensionality  of  the  CS  is  as  large  as 
the  number  of  square -free  monomials  (  2n  )  then  computing  an  orthonormal 
basis  is  problematical.  To  conclude  this  section  it  is  shown  that  for  the 
most  commonly  used  item  response  models,  the  three  parameter  logistic 
models,  J+l  typically  is  equal  to  its  upper  bound. 

Item  response  functions  are  three  parameter  logistic  (3PL)  if 

P.(t)  -  c.  +  (l-c.)[l  +  e'ai(t'bi)]'1 

for  some  item  parameters  a^O,  b^,  and  c^  in  [0,1)  .  It  is  natural  to 
consider  the  item  parameters  random  variables  because  in  most  applications 
they  are  estimated  from  data.  Suppose  the  sampling  distribution  of  the 
estimated  parameters  has  a  continuous  density.  Then  the  following  result  is 
of  interest. 

If  the  joint  distribution  of  the  n  item  parameter  vectors  <a^.,b^,c.>  has 
a  continuous  density,  then  with  probability  one  the  CS  of  the  3PL  item 
response  model  defined  with  sampled  item  parameters  will  have  dimension  2n 

Thus,  for  example,  if  one  begins  with  the  any  published  set  of  estimated 
item  parameters  for  an  application  of  the  3PL  model  and  adds  an  independent 
normally  distributed  "error"  with  zero  mean  and  very  small  variance,  say 
10  to  each  of  the  3n  parameters,  then  with  probability  one  either  one  of 
the  a's  or  c's  will  be  moved  outside  its  allowed  range  or  a  3PL  model  with 
J  as  large  as  it  possible  can  be  will  be  obtained. 
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Proof:  With  probability  one,  the  functions 

a, t  a„t  at 

12  n 

e  ,e  ,  ...  e 

will  be  algebraicly  independent  over  the  reals,  i.e.  will  not  satisfy  any 

nontrivial  polynomial  with  real  coefficients.  But  if  J+l<2n  then  one  of 

the  square-free  monomials  can  be  expressed  as  a  linear  combination  of  the 

remaining  monomials.  On  multiplying  both  sides  of  the  equation  giving  one 

a.t  a.b. 

monomial  as  a  linear  combination  of  the  others  by  positive  n.[e  +  e  ] 

a .  t 

one  obtains  a  polynomial  in  the  e  and  a  contradiction  to  the  hypothesis 
J+l<2n  . 


Section  Four 


Large  Canonical  Spaces 


Consider  Example  Three  for  a  test  with  large  CS  for  an  application  cur¬ 
rently  in  progress.  In  a  large  scale  simulation  we  are  attempting  to 
monitor  and  control  the  changes  in  a  Bayes  modal  ability  estimate  as  new 
items  are  introduced  into  a  100  item  adaptive  test  item  pool.  The  item 
response  function  estimates  for  the  new  items  are  not  expected  to  be  very 
accurate  because  of  motivation,  test  format,  and  ability  distribution  dif¬ 
ferences  between  the  item  response  function  estimation  sample  and  the 
examinees  in  the  application.  The  methods  to  be  reviewed  in  this  section  , 
permit  us  to  compute  as  many  as  we  need  of  the  roughly  2^^  orthornormal 
hj  for  the  test  consisting  of  old  items. 

The  trick  is  to  compute  the  h^  one-at-a- time  in  such  a  way  that  the 
tu  needed  to  complete  the  application  are  computed  first.  Th->s  the  CS  is 


treated  as  the  union  of  nested  vector  spaces  CS 


K 


CSR  -  Span(h0,h1,  ...  hK) 


where  functions  in  only  a  dozen  or  so  spaces  can  be  and  need  be  accurately 
computed.  Some  details  follow. 

A  A 

We  wish  to  approximate  E(0  „,|0-t)  -  R(t)  ,  where  0tn.  is  the  Bayes 

WK.  Wi\ 

mode  adaptive  test  score.  It  turns  out  that  although  J  is  very  large,  the 

A 

projection  R  of  R  into  the  twelfth  space 


R(t)  =  2j£l2  <R,hj>hj (t) 


is  very  close  to  R(t) 


Now  if  q(*;s)  -  hj(s)hj(t)  is  the  projec- 


rQ 

tion  q(*;s)  into  the  twelfth  space  then  Jc  E(0^|  0=t)q(t;s)  dt 


R(t)  .  Thus  if  we  can  write  q(*;s)  as 
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a  linear  combination  of  quasidensities  for  the  K  AR  score  groups  q^X*)  > 
then 

E(,:)  -  Sta;K  ck<s>  Et#MKl*AK  iS  in  Hk]  ' 

The  point  is  that  if  an  application  can  be  completed  using  h^.h^,  .  .  .  h^, 
only  then  it  may  be  possible  to  proceed  as  if  J= 12  . 

This  section  describes  a  general  technique  used  by  our  laboratory  for 
calculating  the  h^  one-at-a-time  in  such  a  way  that  functions  that  are 
likely  to  be  needed  for  an  application  are  well  approximated  by  a  function 
in  CS^  for  small  K  . 

The  General  Method 

The  first  step  of  our  approach  to  large  spaces  is  to  select  a  set  of 
N 

functions  {f  }  ..  that  span  the  CS  and  are  such  that  the  function  of  two 

v  i/-l  v 

variables  2^  f^(s)f^(t)  can  be  easily  evaluated.  For  example  if  f^^, 

...  ...  f  ^  is  any  enumeration  of  the  square -free  monomials  then  the 

f^  span  the  CS .  Furthermore  for  any  s  and  t 

2n  n 

2  f  (s)f  (t)  -  n  [1  +  P.(s)P.(t)] 
v  v  -  11 

i/=l  1 

can  be  evaluated  with  2n-l  multiplications  and  n  additions.  (This  iden¬ 
tity  can  be  verified  by  induction  on  test  length  n  .)  Other  examples  of 
tractible  spanning  sets  and  additional  criteria  for  spanning  sets  are  dis¬ 
cussed  below. 

There  are  two  important  points  to  be  emphasized  here.  Although  there 
are  generally  billions  of  f^  to  enter  into  the  sum  H(s,t)  =  2^  f^(s)ft/(t)  , 
the  multiplicative  formula  for  H(s,t)  requires  only  n  additions  and  2n-l 
multiplications.  Second,  the  ordering  of  the  f^  is  inconsequential. 

Whereas  the  outcome  of  a  Gram-Schmidt  orthogonalization  applied  to  the 
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square-free  monomials  or  any  other  large  set  of  functions  f  would  be  very 
order  dependent,  the  calculation  of  H  is  not. 

The  next  step  in  computing  the  h^  can  be  carried  out  with  commercial 
software  or  can  be  converted  to  a  eigenvalue/eigenvector  problem:  Compute 
positive  numbers  A  and  functions  h  not  identically  equal  to  zero  such 
that  each  h  is  in  the  CS  and  satisfies 


Ah(-)  -  H(» , t)h(t)  dt 


where  H(s,t)  ~  f^(s)f^(t)  .  There  will  be  only  finitely  many  different 

values  of  A  such  that  there  is  some  h^O  in  the  CS  satisfying  the  equa¬ 
tion.  Since  the  h's  are  in  the  CS  there  can  be  only  finitely  many 
linearly  independent  solutions  h  for  any  A  .  Thus  any  maximal  set  of 

linearly  independent  solutions  can  be  subscripted  and  arranged  in  order  of 

their  subscript  so  that  Aa>A.>  ...  >A,,>0  for  some  K<J  and  A.h.(*)  - 

0  1  K  J  J 

J?  H(-,t)h  (t)  dt  . 

^  J 

Without  loss  of  generality  we  can  set  <lu  ,Vk>=*1  since  h^(t)  is  a 

solution  for  A.  if  and  only  if  h. ( • )/<h . ,h.>  is.  Since  the  set  of  all 

J  J  J  J 

h's  corresponding  to  any  A  form  a  vector  space,  they  can  be  selected  to 
be  orthonormal.  Since  it  can  be  shown  h's  with  different  A's  are  or¬ 
thogonal,  the  h.  will  form  an  orthonormal  set  of  vectors  In  fact  it  is 
easy  to  show  that  when  the  f^  span  the  CS,  K=J  and  the  h^  computed  in 
this  way  form  an  orthonormal  basis  for  the  CS.  If  an  application  suggests  a 

set  of  f  that  don't  span  the  CS,  then  K<J  and  the  h.  will  be  a  basis 
"  J 

for  whatever  subspace  the  f  span. 

Note  that  except  in  the  unusual  case  that  more  than  one  h  correpsonds 
to  one  A  ,  the  h's  are  fully  ordered  by  their  A's  .  Even  if  for  some 
j  ,  corresponding  to  different  A's  will  be  ordered  and 

we  can  still  speak  of  Ik  occuring  early  or  late  in  the  sequence  of  h's  . 
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The  ordering  is  important  because  for  various  reasons  (cumulative  numerical 
errors  and  the  fact  that  A^  is  very  close  to  zero  for  large  j  )  the  Vu 
that  occur  early  in  the  sequence  are  relatively  easy  to  compute  (although 
the  remaining  Vu  can  be  very  hard  to  compute) . 

There  are  two  related  advantages  in  arranging  tne  computation  of  basis 
functions  as  described  above.  The  h^  with  large  A^  ,  which  are  easy  to 
compute,  can  be  computed  without  computing  the  tu  with  small  A  ,  which 
can  be  very  hard  to  compute.  This  is  important  because  A^  generally 
measures  the  relative  importance  Ik  in  representing  functions  in  several 
senses.  For  example,  if  f  is  approximated  by  its  projection  into 

A 

CS^-spanOiQ,  ...  h^)  ,  which  turns  out  to  be  f^(  • )  =  2  <f^,h.>h.  for 

v  j<K  ^  J  J 

K<J  ,  then  the  total  error 

s  J  (f/t)  -  yt)]2  dt 

V 

is  simply  2  A.  .  (This  sum  can  be  evaluated  even  if  J  is  very  large 


because 


Jc  H(t, t)  dt 


2  A. 


2  A.  .) 


As  a  bonus,  the  method  also  delivers  a  set  of  statistics  X.  needed 

J 

for  Example  One  and  Theorem  One  (i.e.,  statistics  such  that  ■* 

E[X.|0*=t]  for  all  t  in  [c,d]  ).  Details  are  given  in  the  final 
subsection. 


Some  Examples  of  Spanning  Sets 


In  addition  to  the  square- free  monomials  we  use  the  2  likelihood 


functions  for  short  tests.  Here 
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k  k 

n  u.  (1-u.  ) 

f„(t)  =  n  p  (t)  1,l/  [l-p.(t)]  L,l/ 

i=l  1 

where  u^,  ...  u^,  ...  u  ^  is  any  enumeration  of  the  2n  item  response 
patterns.  For  these  functions 


H(s,t)  -  S  f  ( s ) f  (t) 

1 /  J/  1/  ' 


n 


“  n  (P.(s)p.(t)+(i-p.(s)](i-p.(t)]}  , 

which  can  be  easily  evaluated.  (This  also  can  be  proven  by  induction  on 
test  length  n  after  noting  that  each  likelihood  function  can  be  written  as 
n  *  * 

f  (t)  «*  II  {u.  P.(t)  +  (l-u.  )[1-P.(t)l)  .)  These  functions  certainly 

v  .  -  1,1/  1  i,i//l  i'  J 

i=l  ’ 

span  the  CS  because  any  square- free  monomial  can  be  written  as  a  linear  com¬ 
bination  of  likelihood  functions.  (To  prove  this,  simply  write  the  general 

k 

monomial  II  P.  as  the  sum  of  the  likelihoods  for  patterns  u  with 
j-r  X3 

k  k  k 

U.  “U.  «*  ...  U.  “1  .  ) 

X1  12  1r 

For  adaptive  tests  and  long  tests  satisfying  (exactly  or  approximately) 
an  algebraic  property  described  below,  we  use  likelihood  functions  for 
selected  subtests.  For  example  to  study  a  fixed  length  adaptive  test  of  15 
items  with  a  100  item  pool  it  is  natural  to  consider  the  ^5]  <<: 
likelihood  functions  with  fifteen  factors  since  every  statistic  computed 
from  an  examinee's  score  depends  on  only  15  item  scores. 

The  discussion  of  the  Rasch  model  introduces  a  second  rationale  for 
forming  the  f^  from  the  likelihood  functions  for  short  subtests.  Recall 
that  for  the  Rasch  model  every  polynomial  in  the  CS  could  be  rewritten  as  a 
"polynomial"  in  the  CS,  no  monomial  of  which  contained  2  or  more  factors. 
This  property  is  remarkably  general.  For  the  3PL  model  (and  most  of  its 
generalizations)  every  polynomial  in  the  CS  can  be  rewritten  as  a  linear 
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combination  of  monomials  with  five  or  fewer  factors,  at  least  to  a  surpris¬ 
ingly  high  degree  of  approximation^. 

When  every  function  in  the  CS  can  be  expressed  as  a  linear  combination 
of  square -free  monomials  with  five  or  fewer  factors,  then  the  CS  is  spanned 
by  the  likelihood  functions  from  subtests  with  five  factors.  There  are 

still  an  enormous  number  of  likelihood  functions  f  that  can  be  formed 

u 

from  from  all  five  item  subtests.  Nonetheless  H(s,t)  ■=>  2  f^(s)f^(t)  can 
be  computed  efficiently  for  these  functions  as  follows: 


Let  F^(s,t)  abbreviate  P^(s)P^(t)  +  (l-P^(s) ] (l-P^(t) ]  . 

Let  H^Cs.t)  denote  the  sum  of  the  likelihood  functions  for  all  i 

i 

item  subtests  formed  from  the  first  m  items. 

To  initialize  set 


Hj(s,t)  -  F1(s,t) 


iL(s,t)  ”  0  for  i=2 , 3 , . . . 5  . 


To  update ,  compute 


H™+1  -  F  -H™  .  for  i=2 , . . . 5 
i  m+1  l-l 


m 


Hm+1  -  F  +  H' 

1  m+1  1 

If  in  the  update  step  is  computed  first,  followed  by  H™+  ,  etc., 

then  can  be  written  over  H™  and  the  amount  of  storage  required  by 

the  algorithm  can  be  kept  small. 

Most  of  our  current  applications  to  one  dimensional  ability  tests  use 


this  algorithm.  Although  some  of  the  CS  may  be  left  out,  the  algorithm  in 
practice  works  very  well.  It  is  the  only  algorithm  that  has  consistently 
produced  useful  results  with  long  tests. 
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Reduction  of  Proofs  to  Matrix  Algebra 

A  number  of  assertions  were  made  without  proof  concerning  the  solutions 
for  the  functional  equation 

0(h)  -  Ah 

where  0(h)(*)  -  H(*,t)h(t)  dt  for  H(s,t)  =  2^  f^sjf^t)  . 

By  taking  advantage  of  the  finite  dimensionality  of  the  CS  these  proofs  can 

be  obtained  with  matrix  algebra.  In  this  section  the  reduction  to  matrix 

algebra  is  indicated  after  a  few  of  the  assertions  are  proven  directly. 

First  0  is  a  transformation  of  the  CS  to  itself  because  the  f  are 

v 

in  the  CS  and  0(h)  “2  <f  ,h>f  is  a  linear  combination  of  the  f  .  0 

v  v  v  v 

is  thus  a  linear  mapping  of  a  finite  dimensional  vector  space  into  itself. 

To  show  that  the  eigenfunctions  of  0  span  the  CS  it  is  neccesary  to 
show  that  0  maps  the  CS  onto  the  CS.  Equivalently,  since  the  CS  is  finite 
dimensional,  one  may  show  0(h)~O  implies  h-=0  .  To  show  this  one  can 

J  j 

write  f  (•)  =  2  a  . g.(*)  for  some  orthonormal  basis  (g.).  „  .  The 

"  j„0  ‘'JTJ  6J  J“0 

matrix  A °(a  .)  must  have  rank  J+l  since  the  f  span  the  CS.  If 

0(h)=O  ,  then  0  ■=  <g^ ,0(h)>  =  e?A^A<g,h>  ,  j=0,...J  where  e^  is  the  jth 

unit  vector  and  <g,h>  is  the  column  vector  of  <g.,h>'s  .  Thus 
T  T 

A  A<g,h>«0  .  Since  A  A  has  rank  J+l  ,  <g,h>=0  ,  i.e.,  h  is  orthogonal 

to  each  g.  .  Thus  h=0  . 

J 

The  existence  of  eigenfunctions  in  the  CS  and  the  fact  that  the  eigen¬ 
functions  span  the  CS  can  be  shown  with  matrix  algebra.  To  introduce  matrix 
notation,  for  each  t  in  [c,d]  let  f(t)  be  the  column  vector  with  vth 
coordinate  f  (t)  .  Then  H(s,t)  is  the  scalar  product  of  f(s)  and 
f(t)  .  Let  Q  denote  the  matrix  of  definite  integrals 
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Q  -  r  f(t)fA(t)  dt  ,  i.e.  Q  is  the  matrix  with  typical  entry  q  ,  - 
c  i/v 

<f  ,f  ,>  . 

v  v' 

Q  must  be  positive  definite  or  positive  semidefinite  since  for  any 
T  d  2 

vector  a  ,  a  Qa  =  Jc  [a*f(t)J  dt  >  0  .  Therefore  for  some  K  ,  Q  can  be 
0  1  K  T  0  1  K 

written  Q  -  [a  ,a  ,  ...  a  ]  D[a  ,a  ,  ...  a  ]  for  K+l  orthonormal  vectors 
a~*  and  a  diagonal  matrix  D  having  positive  diagonal  entries  dj>0  •  F°r 
0<j<K  let  h.  be  defined  by 

hj (t)  -  d‘1/2  aj-f(t)  . 

Since  each  h.  is  a  linear  combination  of  functions  in  the  CS ,  each  must  be 

in  the  CS.  The  h.  are  orthonormal  since 

J 

<hj,hk>  -  d‘1/2dk'1/2  ajT  Jjj  f(t)fT(t)  dt  ak 

-  *]1/2\1/2  .jV  . 


TO  ,  if  j; 

U  .  if  J* 


In  fact  the  h.  must  be  eigenfunctions  of  i{>  because 

^(h  )  -  J*  fT(t)f(.)d:1/2  aj-f(t)  dt 
J  °  J 

“  d"1/2  ajT  Jjj  f(t)fT(t)  dt  f(-) 


d-l/2  ajT  ) 

J 

.1/2  jT,.  . 
d.  aJ  f (• ) 

J 


d.h.  . 

J  J 


K  must  equal  J  because  otherwise  $  would  not  map  the  CS  onto  the  CS. 
Thus  the  eigenfunctions  form  an  orthonormal  basis  for  the  CS. 
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The  Statistics  X^ 

In  Example  One  and  Theorem  One  statistics  with  regression  functions 
equal  to  Ik  were  needed.  Of  course  such  statistics  exist  because  every 
function  in  the  CS,  by  definition,  is  the  regression  function  of  at  least 
one  statistic.  Finding  a  statistic  matching  a  function  fortunately  turns 
out  to  be  easy  for  bases  formed  from  eigenfunctions. 

When  the  Ik  are  obtained  as  eigenfunctions,  these  statistics  are  cal¬ 
culated  in  two  steps.  First,  the  examinee's  data  is  transformed  into  a 
continuous  function  X(t)  .  Then  a  statistic  is  obtained  by  computing 

<X,h.>/A .  . 

J  J 

For  concreteness  consider  the  second  example  of  the  general  method  in 
which  each  f^  is  a  likelihood  function.  The  general  technique  applied  to 
this  example  gives  X(t)  equal  to  the  familiar  likelihood  function  as  the 
random  function 

n 

X(t)  -  n  [u.P.(t)  +  (l-u.)Q.(t)] 
i=l  1 

and  Xj  “  X(t)h.(t)  dt/A^  . 

To  verify  that  the  regression  function  for  this  statistic  is  Ik  ,  we 
compute  as  follows.  The  regression  function  for  X^  evaluated  at  O'* s  is 

E[X.\0=s]  -  A’XE(  JX(t)hj(t)  dt|0=s] 

1  n 

=  a:1  J  n  '  [P  (s)P.(t)  +  Q.(S)Q  (t)]h.(t)  dt 

J  i.  i.  x.  i  j 

=  Aj1  J  H(s,t)h  (t)  dt 
-  lys)  . 

The  general  rule  for  obtaining  a  random  function  X(t)  for  arbitrary 
f  is  to  make  the  replacements 


P1(s)  -»  u]L 
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P2(S)  -  U2 


in  f  (s)  to  obtain  a  random  variable  Y  (u)  from  f  (s)  .  A  random 

V  V  V 

function  X  is  defined  by 


X(t)  =  2  Y  (u)f  (t)  . 

V  V  v 

Finally,  a  random  variable  having  regression  function  equal  to  the  jth  basis 


function  is  obtained  as  J  X(t)h.(t)  dt/A.  .  To  summarize 

c  J  J 


Let  Hfs.t')  -  2  f  (s)f  ft1)  for  functions  in  the  CS  f  not  necessarily 

“ -  '  »  •  jy  '  •  1/  '  ' - - - - -  1/  - ■*“ 

spanning  the  CS.  Let  h  satisfy  H(*,t)h(t)  dt  -  Ah(*)  for  positive 
A  .  For  each  t  in  [c,d]  let  X(t)  be  the  random  variable  obtained  by 


replacing  each  P^(s)  by  u^  in  the  formula  defining  H(s,t)  .  If  Xj  ° 

<X,h.>/A .  ,  then  E[X.|0-t]  =  h.(t)  for  c<t<d  . 

J  J  J  J 


Note,  the  transformation  f^(s)  -»  Y^  generally  cannot  be  defined  on  the  CS 
because  if  two  items  have  the  same  item  response  function,  then  we  can  have 
f^(  • )  ”  f^,(*)  as  functions  in  the  CS  but  Y^Y  ,  .  The  problem  can  be 
avoided  by  regarding  f^(s)  as  a  polynomial  with  real  coefficients  in 
algebraicly  independent  variables  P^(s) ,  ,  ...  P  (s)  • 


Proof:  E[X(t)  |  0=s]  -  H(s,t)  . 


NOTES 


1.  Levine,  M.  and  Williams,  B.  Latent  trait  theory  as  fundamental 

measurement ,  Paper  presented  at  Society  for  Mathematical  Psychology 
Annual  Conference,  Irvine,  California,  1989. 


page  47 


Acknowledgements 

This  research  was  supported  by  the  Cognitive  Science  Program  of  the 
Office  of  Naval  Research  contract  N00014-83K-0397 ,  NR  150-518  and  N00014- 
86K-0482,  NR  4421546.  I  am  indebted  to  Bruce  Williams,  Tim  Davey,  Charles 
Davis,  Fritz  Drasgow,  Brian  Junker,  and  Gary  Thomasson  for  detailed  comments 
on  earlier  versions  of  this  work.  Conversations  with  J.O.  Ramsay  were  also 
useful . 


1583/08/04 


1893/08/04 


University  of 


Or.  ferry  Ackerman 
Educational  Psychology 
210  education  Old). 

Umverci  ty  of  111  inou 
Chjnpaign,  It  Cl 001 

Or.  Robert  Ahler* 

Code  H7II 

Hunan  factors  Laboratory 
Naval  Training  System  Center 
Orlando,  FL  32013 

Or.  Janes  Algma 
M03  No  man  Halt 
University  of  Florida 
Gainesville,  fL  32605 

Or.  Crling  0.  Andersen 
Department  of  Statistics 
Studiestraede  G 
1 455  Copenhagen 
OCNNARK 

Or.  Eva  l.  OaVer 
UCLA  Center  /or  the  Study 
of  Evaluation 
145  Moore  Hal  I 
University  of  California 
Lot  Angeles,  CA  50024 

Or.  Isaac  Oejar 
Hail  Stop:  10-R 
Educational  Testing  Service 
Rosedale  Road 
Princeton,  NJ  0054 1 

Or.  Menuchs  Oirenbau* 

School  of  education 
Tel  Aviv  University 
Ra#at  Aviv  69970 
ISRAEL 

Or,  Arthur  S.  Blaiwts 
Code  N7I2 

Naval  Training  Systems  Center 
Orlando,  FL  32013-7100 

Or.  Bruce  Blo*om 

Defense  Manpower  Oata  Center 

99  Pacific  St. 

Suite  155A 

Monterey,  CA  93943-3231 

Or.  Stanley  Collyer 
Office  of  Naval  Technology 
Code  222 

600  N.  Quincy  Street 
Arlington,  VA  22217-5000 

Or.  Hans  F.  Crombag 
Faculty  of  Law 
University  of  Limburg 
P.O.  Be*  $16 
Maastricht 

The  NETKRLANOS  6200  MO 

Or.  Timothy  Oavey 

American  College  Testing  Program 

P.O.  Bo*  168 

Iowa  C.ty,  !A  52243 

Or.  C.  M.  Cayton 
Department  of  Measurement 
Statistics  t  Evaluation 
College  of  Education 
University  of  Maryland 
College  Park,  MO  20742 

Or.  Ralph  J,  Or Ay a  I  a 
Measurement,  Statistics, 
and  Evaluation 
Benjamin  Bldg. ,  Rm.  4112 
University  of  Maryland 
College  Park,  MO  20742 

Or.  Oattprasad  Oivgi 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.O.  8o«  16268 
Alemandna,  VA  22302-0268 

Or.  Hei-Ki  Oong 
Bell  Communications  Research 
6  Corporate  Place 
PYA-1K226 

Piscatiway,  NJ  06854 

Or.  F nt:  Orasgow 
University  of  Illinois 
Department  of  Psychology 
E03  E.  Oamel  St. 

Champaign,  IL  GI020 


1 1 1  inois/lcvmc 


Um vc rs« t»  of 


Or.  R.  Darrell  Dock 
University  of  Chicago 
fiORC 

G030  South  Ell  is 
Chicago,  IL  60637 

Cdt.  Arnold  Ochrer 

Sectie  Psychologiseh  Onderxoek 

ReVruter ings-Cn  Se I ec t« ecenlrwn 

Kwartier  Keningen  Ast-id 

Brui jnstraat 

1120  Brussels,  BELGIUM 

Or.  Robert  Greau* 

Code  70 

Naval  Irainmn  Systems  Center 
Orlando,  Fl  32613-7100 

Or.  Robert  Brennan 
American  Col  lege  testing 
Programs 
P.  0.  Bo*  168 
Iowa  City,  1A  52243 

Or.  John  B.  Carrol l 
<09  Elliott  Rd.,  North 
Chapel  Hill,  NC  27514 

Dr.  Robert  M.  Carrol  I 
Chief  of  Naval  Operations 
0P-0I02 

Washington,  OC  20350 

Or.  Raymond  E.  Christal 
UES  LAMP  Science  Advisor 
ATHRL/MOCL 

Brooks  AFB,  TX  78235 

Or.  Norman  Cliff 
Department  of  Psychology 
Univ.  of  So.  California 
Los  Angeles,  CA  90009-1061  x 

Director , 

Manpower  Support  and 
Readiness  Program 
Center  for  Naval  Analysis 
2000  North  Beauregard  Street 
Alexandria,  VA  22311 


Oeftnse  Technical 

Information  Center 
Cameron  Station,  Bldg  5 
Alexandria,  VA  22314 
Attn:  IC 
<12  Copies) 

Or.  Stephen  Ounbar 
22<B  Lindgu»st  Center 
for  Measurement 
University  of  Iowa 
lowj  City,  lA  52242 

Or.  James  A.  Earles 

Air  Force  Human  Resources  Lab 

Orooks  AF0,  TX  70235 

Or.  Kent  Eaton 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Or.  Susan  Cmbrctson 
University  of  Kansas 
Psychology  Department 
426  Fraser 
Lawrence,  KS  660<5 

Or.  George  Englehard,  Jr. 

Oi vi sion  of  Educational  Studies 
Emory  University 
210  Fishburne  Bldg. 

Atlanta,  GA  30322 

Or.  Benjamin  A.  fairbank 
Performance  Metrics,  Inc. 

5825  Cal laghan 
Suite  225 

San  Antonio,  TX  70228 

Or.  P-A.  Federico 

Code  Si 

NPROC 

•San  Oiego,  CA  921S2-6000 

Or.  Leonard  Feldt 
Lindquist  Center 
for  Measurement 
University  of  Iowa 
Iowa  City,  1A  52242 


Or.  Richard  L.  Ferguson 
Amer ican  Col  lege  Testing 
P.O.  Be*  IGO 
lewa  City,  IA  52243 

Dr.  Gerh.vd  Fischer 
Liebiggasse  5/3 
A  1010  Vienna 
AUSTRIA 

Or.  Myron  F ischl 
U.S.  Army  Headquarters 
OAPC-MPR 
The  Pentagon 

Washington,  DC  20310-0300 

Prof.  Oonald  Fitzgerald 
University  of  New  Cngland 
Department  of  Psychology 
Armidale,  New  South  Wales  2301 
AUS1RAL1A 

Mr.  Paul  Foley 

Navy  Personnel  R&O  Center 

San  Oiego,  CA  92152-6000 

Or.  Alfred  R.  F regly 
AFOSR/NL,  Bldg.  410 
Bolling  AFB,  DC  20332-6446 

Or.  Robert  0.  Gibbons 
Illinois  State  Psychiatric  Inst. 
Rm  S29W 

1601  U.  Taylor  Straet 
Chicago,  IL  60612 

Or.  Janice  Gifford 
University  of  Massachusetts 
School  of  Education 
Amherst,  KA  01003 

Or.  Robert  Glaser 
Learning  Research 

&  Development  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  IS260 

Or.  Bert  Green 
Johns  Hopkins  University 
Oepart«ent  of  Psychology 
Charles  l  34th  Street 
Baltimore,  MO  21210 


Mr.  Dick  Hoshaw 

OP- 1 35 

Ar  I  mgton  Anne* 

Room  2834 

Washington,  OC  20350 

Or.  Lloyd  Humphreys 
University  of  Illinois 
Department  of  Psychology 
603  East  Oaniel  Street 
Champaign,  IL  61020 

Or.  Steven  llunka 
3-104  Educ.  N. 

University  of  Alberta 
Edmonton,  Alberta 
CANAOA  T6G  2C5 

Or.  Huynh  Huynh 
College  of  Education 
Univ.  of  South  Carol ma 
Columbia,  SC  29206 

Or.  Robert  Jannaront 
Cite,  and  Computer  Eng.  Dept. 
University  of  South  Carol  me 
Columbia,  SC  29200 

Or.  Douglas  M.  Jones 
Thatcher  Jones  Associates 
P.O.  Bo*  6640 
10  Trafalgar  Court 
Lawrencevi I le,  NJ  006<6 

Or.  Brian  Junker 
University  of  Illinois 
Department  of  Statistics 
101  lllim  Hall 
725  South  Wright  St. 
Champaign,  IL  61020 


Or.  Mi  lion  S.  Kali 
European  Science  Coordination 
Office 

U.S.  Army  Research  Institute 
Bo*  6S 

FPO  New  York  09510-1500 

Prof.  John  A.  Keats 
Department  of  Psychology 
University  of  Newcastle 
II.S.W.  2300 
AUSTRALIA 


1 1 1  mois/Levine 


OOPSIEP  0*6*4 

P.O.  0c«  H20 

0-7590  f  nedr »<hshafen  i 

wCSI  GERMANY 

Prof.  Edward  l«aer tel 
School  of  Education 
Stanford  University 
Stanford,  Ca  94305 

Dr.  Ronald  K.  Hamblelon 
University  of  Massachusetts 
Laboratory  of  Psychometric 
and  Evaluative  Research 
Hi  I  Is  South,  ficoo  152 
Amherst,  MA  01003 

Or.  Oelwyn  Harnisch 
University  of  Illinois 
51  Gerty  Drive 
Champaign,  IL  61020 

Or.  Grant  Henning 
Senior  Research  Scientist 
Oivision  of  Measurement 
Research  and  Services 
Educational  Testing  Service 
Princeton,  NJ  00541 

Ms.  Rebecca  Hetter 
Navy  Personnel  R&O  Center 
Code  63 

San  Oiego,  CA  92152-6800 

Or.  Paul  W.  Holland 
Educational  Testing  Service,  21-T 
Rosedale  Road 
Princeton,  NJ  08541 

Prof.  Lut*  F.  Hornke 
Institut  fur  Psychologic 
RwlH  Aachen 
Jaegerstrasse  17/19 
0-5100  Aachen 
wt$l  GERMANY 

Or.  Paul  Horst 
677  G  Street,  #104 
Chula  Vista,  CA  92010 


Or.  G.  Gag#  Kingsbury 

Portland  Public  Schools 

Research  and  Evaluation  Department 

501  North  Oi»on  Street 

P.  0.  Bo.  3107 

Portland,  OR  97209-3107 

Or.  William  Koch 
Bo*  7246,  Meas.  and  Evai.  Ctr. 
University  of  Ie*as-Austin 
Austin,  TX  70703 

Or.  Leonard  Kroeker 
Navy  Personnel  R&O  Center 
Code  62 

San  Oiego,  CA  92152-6800 

Or.  Jerry  LehnuS 

Defense  Manpower  Oata  Center 

Suite  400 

1600  Wi Ison  Bl vd 

Rosslyn,  VA  22209 

Or.  Thomas  Leonard 
University  of  Wisconsin 
Department  of  Statistics 
1210  West  Oayton  Street 
Madison,  w|  53705 

Or.  Michael  Levine 
Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champa ign,  U  GlCOl 

Dr.  Charles  lewis 
Educational  Jesting  Service 
Princeton,  NJ  06641-005: 

Dr ,  Rober  l  L .  t  mn 
Campus  0e>>  2<9 
University  of  Colorado 
Boulder,  CO  80309-0249 

Or.  Robert  loefcman 
Center  for  Naval  Analysis 
4401  F ord  Avenue 
P.O.  Oo«  16260 
Alexandria,  VA  22302-0260 

Or,  Frederic  M,  lord 
Educational  Testing  Service 
Princeton,  NJ  00541 


1089/08/04 


I  989/08/0*. 


Urn  i  tv 


Or.  George  0.  Macready 
Oepartnent  of  Measurement 
Statistics  &  Evaluation 
Col lege  of  education 
Umvtrtity  of  Maryland 
C'lUg*  Park,  MO  2074? 

Or.  Cary  Marco 
Slop  31-C 

Ed-cational  Testing 
Princeton,  NJ  00491 

Or,  James  R,  McBride 
The  Psychological  Corporation 
1250  $i'  th  Avenue 
San  Oiego,  CA  92101 

Or,  Clarence  C.  HeCormd 
HQ,  USKCPC0M/MCPC1 
2S00  Green  Oay  Road 
North  Chicago,  II  60064 

Mr,  Christopher  HcCusker 
Univtrti ty  of  Illinois 
Oepart«eM  of  Psychology 
G03  C.  Oan.el  St. 

Champaign,  It  G1020 

Or,  Robert  McKinley 

taw  School  Admission  Service! 

Do*  40 

t.’ewtvwn,  PA  109 »0 

Or.  Janet  McHichael 
Technical  Oirector 
Navy  Personnel  RSD  Center 
San  Oiego,  CA  92152-6000 

Hr.  Alan  M«ad 
c/o  Or.  Michael  Levin# 
educational  Psychology 
210  education  Oldg. 

University  of  Illinois 
Champaign,  It  GlOOl 

Or.  Robert  Mitlevy 
educational  Testing  Service 
Princeton,  MJ  0054< 

Or ,  Kill, an  Montague 

Wfifc  Code  13 

<an  Diego,  CA  92I52-GC00 


Or.  Janes  0.  Olsen 
KICAT  Systems 
1075  South  State  Street 
Oren,  UT  G<058 

Office  of  Naval  Research, 

Code  1M2CS 
800  N.  Quincy  Street 
Arlington,  VA  22217-5000 
<6  Copies) 

Off  »ct  of  Naval  Research, 

Code  125 

000  M,  Quincy  Street 
Arlington,  VA  22217-5000 

Assistant  for  MPT  Research, 
Development  and  Studies 
OP  0107 

Washington,  DC  20370 

Or.  Jwdi th  Oratanu 
Oasic  Research  Office 
Amy  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Or.  Jets#  OrlansVy 
Institute  for  Oefense  Analyses 
1001  M.  Oeauregard  St. 
Alexandria,  VA  22311 

Or,  Peter  J.  PasMey 
educational  Testing  Service 
Rosedale  Road 
Princeton,  NJ  00541 

Wayna  M,  Patience 
American  Council  on  education 
G£D  Testing  Service,  Suite  20 
One  Oupont  Circle,  fftf 
Washington,  OC  20036 

Dr.  Janes  Paulson 
Oepartnent  of  Psychology 
Portland  State  University 
P.0.  Oo«  751 
Portland,  OR  37207 

Oept.  of  Adnims trati ve  Sciences 
Code  54 

Naval  Postgraduate  School 
Monterey,  CA  9390-5026 


>f  II I mois/Ltvane 


University  <f 


Ms,  Kathleen  Moran© 

Navy  Personnel  RiO  Center 
Code  62 

San  O.ego,  CA  92152-6000 

Headquarters  Marine  Corps 
Code  M*'|-20 
Wvsi.xngun,  DC  20380 

Cr.  Ratna  Nandakw*ar 
Dept,  of  educational  Stud*es 
Willard  Hall ,  Roc*  213 
University  of  Delaware 
Newjri,  DC  19716 

Or.  U.  Alan  Niccwander 
University  of  Oklahoma 
Oepartnent  of  Psychology 
Norman,  OK  73071 

Oeputy  Technical  Oirector 

l.f'ROC  Code  01 A 

San  Oiego,  Ca  92152-6000 

Director,  Training  Laboratory, 
NPROC  (Code  05) 

San  Diego.  CA  92152-6000 

O'rector,  Manpower  and  Personnel 
laboratory, 

NPROC  (Code  0G> 

San  Diego,  CA  92152-6000 

Oirector,  Hyman  factors 

l  Organisational  System  Lab, 
NPROC  (Code  07) 

San  O.ego,  CA  92152-6000 

Library,  NPROC 
Code  P20IL 

San  O.ego,  CA  92152-6000 

Commanding  Officer, 

Naval  Research  Laboratory 
Code  2C27 

Washington,  OC  20390 

Or.  Harold  T.  O'Neil,  Jr. 

School  of  Education  -  W?H  001 
Oepartnent  of  educational 
Psychology  l  Technology 
University  of  Southern  California 
Lcs  Angeles.  CA  90009-0031 


Oepartnent  of  Operations  Research. 

Naval  Postgraduate  School 
Monterey,  CA  93940 

Or.  Mark  0.  RecVase 
ACT 

P.  0.  Oo»  160 
Iowa  City.  I A  52243 

Or.  Malcolm  Ree 
AfHRl/MOA 

Brooks  AfB,  lx  70235 

Mr.  Stmve  Reiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  C.  River  Road 
Minneapolis,  MM  55455-0344 

Or.  Carl  Ross 
CNCT-POCO 
Bui Iding  90 

Great  Lakes  NIC,  IL  60000 
Or.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Cclunbia,  SC  29200 

Or.  fumko  Samfjma 
Oepartnent  of  Psychology 
University  of  Tennessee 
3100  Austin  Peav  Bldg. 

Knoxville,  IN  37916-0900 

Mr.  Orew  Sands 
NPROC  Code  62 
San  O.ego.  Ca  921S2-GOOO 

l owe  1 1  Schoer 

Psychological  &  Quantitative 
foundations 
College  ef  Education 
University  of  Iowa 
Iowa  City,  I A  52242 

Or.  M.ary  Schrati 
905  Orchid  Way 
Carlsbad,  CA  92003 


Or  rf,  Steve  Selinan 
PA;tM*»ftAU> 

2P2*'9  The  Pentagon 
Washington,  OC  20301 

Or.  I ajvo  Shigenasu 
7-5-24  Kuger.u"*-Kaigan 
fv,tis>-a  2M 
JAPAN 

Or.  W.  1 1 1 an  Sms 
Center  for  Naval  Analysis 
4401  ford  Avenua 
P.O.  On.  16260 
Alexandria,  VA  22302-0260 

Or.  H.  Wallace  Smaiko 
Manpower  Research 

and  Advisory  Services 
Smthsonian  Institution 
001  North  Pitt  street.  Suite  120 
Alexandria,  VA  22314-1713 

Or.  Richard  C.  Snow 
School  of  Education 
Stanford  University 
Stanford,  Ca  94305 

Dr.  Richard  C.  Sorensen 
Navy  Personnel  RiO  Center 
San  O.ego,  CA  92152-6000 

Dr ,  Judy  Spray 

ACT 

P.C  Oo.  160 
lo-a  C«ty,  1A  52243 

Or.  Martha  Stocking 
Educational  Testing  Service 
Princeton,  NJ  00541 

Or.  Peter  Stoloff 
Center  for  Naval  Analysis 
4401  ford  Avenue 
P.O.  0ox  16260 
Ale-andna.  VA  22202-0260 

Or.  Wi Ilian  Stout 
University  of  Illinois 
Oepartnent  of  Statistics 
101  1 1  i  mi  Hall 
725  South  Wright  St. 

Champaign,  IL  61620 

Or.  ledyard  Tucker 
Univers*  ty  of  III  mois 
Oepartnent  of  Psychology 
603  E.  Daniel  Street 
Champaign,  IL  61820 

Or.  Oavid  Vale 
Assessment  Systens  Corp. 

2233  University  Avenue 
Suite  4<0 

St.  Paul.  MN  551 1 4 

Dr,  fnrl  L.  VieiAO 
Navy  Personnel  RfcO  Center 
San  o.ego,  Ca  92152-GOOO 

Or.  Howard  Warner 
Educational  Testing  Service 
Princeton,  NJ  08S4I 

Or.  Mmg-Mei  Wang 
l  mdquist  Center 
for  Measurement 
University  of  Iowa 
Ixwa  C.ty.  IA  52242 

Or.  TNcmas  A.  Warn 
f AA  Academy  AAC9340 
P.O.  Oo.  2S002 
Oklahoma  City,  CK  73125 

Or.  Orian  Waters 

HunRRO 

12900  Argyle  Circle 
Alexandria,  VA  22314 

Cr.  David  J.  Wens 
N6C0  CU*ott  Hall 
University  ef  Minnesota 
"5  E.  River  Road 
Minneapolis,  MM  55455-0344 

Or.  Ronald  A,  Weitjnan 
Oo*  |4& 

Carmel,  CA  93921 

Major  John  Welsh 
AfHRt/MOAU 

Brooks  AfO.  TX  70223 


S 1 1  moi s/l ev«ne 


Dr.  Ilar.earan  C»«\m.r.Al Nan 
laboratory  nf  f’Sy(hnnttr  «<  and 
tva»v*ti<o  S-esea'Ch 
Schcol  Cr  fdvcat»cr 
University  of  Massachusetts 
AmKerst,  ma  OIC32 

*r,  0r**5  Sv*PS''- 

Navy  Personnel  RtO  Center 

Code-131 

San  O.ego,  Ca  32152-6800 

Or.  John  Tangney 
AfGSR/M,  &idg.  4 10 
Bollmg  Af&,  CC  20232-6440 

Or.  Kikuni  Tatsucka 
CERL 

252  Cngmeer«ng  Research 
Laboratory 

103  $.  Mathews  Ax-nue 
Urbana,  IL  61C0I 

Or.  Maurice  TatSwO*! 

220  Education  Oldg 
1310  S.  S..th  S‘. 

Champaign,  IL  61020 

Or.  Oavid  "hissen 
Department  of  Psychology 
University  of  Kansas 
Lawrence,  > S  66044 

Mr.  lhonas  J.  Thomas 
John*  llopk.ns  University 
Oepartnent  ef  Psychology 
Charles  ft  34lh  Street 
Baltimore,  MO  21216 

Mr.  Gary  Thonasson 
University  of  Illinois 
educational  Psychology 
Champaign,  li  61020 

Or.  Robert  TsutaVawa 
University  of  Missouri 
Oepartnent  of  Statistics 
222  Hath.  Sciences  Bldg. 
Columbia,  MO  652 M 


Dr.  Oouglas  Wetiel 
Code  51 

Navy  Personnel  RftD  Center 
San  Oiego,  CA  92152-6000 

Or.  Rand  R.  Wi leo» 

University  of  Southern 
Cal  iforma 

Oepartnent  of  Psychology 
lop  Angeles.  C*  90063-1C61 

German  Military  Repr esenta t i ve 
a!  til;  Wolfgang  Wi  Idgrilbe 
Streitiraef teamt 
0-5200  Oenn  ? 

4009  Brandywine  Street,  NV 
Washington,  DC  200|f 

Or,  Bruce  Williams 
Oepartnent  of  Educational 
Psychology 

University  of  Illinois 
Urbana,  K  61801 

C’r .  Hi  Ida  W«ng 

NRC  MM* 1 76 

2101  Constitution  Ave. 
Washington,  OC  20419 

Mr.  John  M.  Wolfe 

Navy  Personnel  RftO  Center 

San  O.ego,  CA  92152-6800 

Or.  George  U©ng 
Omstatistics  Laboratory 
Memorial  $loan-Ketter mg 
Cancer  Center 
1275  York  Avenue 
New  York,  NY  10021 

Or.  Wallace  Wulfeek,  III 
Navy  Personnel  RftO  Center 

Code  51 

Sen  O.ego.  CA  92IS2-6D00 

Dr.  Kentaro  Yamamoto 
03- T 

Educational  Testing  Service 
Rosedale  Road 
Princeton,  NJ  O0S41 


Or,  0*n  Segal  I 

Navy  Personnel  RftO  Cente 

San  Diego,  CA  92152 


1 393/0$/ 0 


U-Mvtrs«l>  of 


Or.  Wendy  V<n 
CJ0/«cCf»-  H.n 
D< l  tionle  Research  Part 
Monterey,  CA  939*0 

Or.  Joseph  l.  ^oung 
National  Science  foundation 
Reon  320 

IOOO  G  Street,  N.w. 
Washington,  OC  20SS0 

Mr.  Anthony  R.  2ara 
National  Council  of  State 
Qoards  of  Nursing,  Inc. 
025  North  Michigan  Avenue 
Suite  1S<* 

Chicago,  II  60611 


